<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Icaro on Medium]]></title>
        <description><![CDATA[Stories by Icaro on Medium]]></description>
        <link>https://medium.com/@icaro_vazquez?source=rss-7d0c8410b20b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2WgZBhbDThT22zo96X3CkA.jpeg</url>
            <title>Stories by Icaro on Medium</title>
            <link>https://medium.com/@icaro_vazquez?source=rss-7d0c8410b20b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:26:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@icaro_vazquez/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Healthy Network & Broken Calls: Teaching an Agentic AI Investigator to Tell Trigger from Root Cause]]></title>
            <link>https://medium.com/@icaro_vazquez/healthy-network-broken-calls-teaching-an-agentic-ai-investigator-to-tell-trigger-from-root-cause-a7322e5f933c?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7322e5f933c</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[root-cause-analysis]]></category>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 00:01:17 GMT</pubDate>
            <atom:updated>2026-11-03T20:32:01.608Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Building an Enterprise Agentic AI Investigation Team: Adding a Second Scenario Where the Network Looks Fine</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MVSMBcwpwNAdc48kPRsGfw.png" /></figure><h3>TL;DR</h3><p>I added a 2nd scenario to my Agentic AI Investigation Team: one where the problem is harder to find, and the network infrastructure shows every link is up, yet video calls break. This 2nd scenario broke our investigation team in many ways. Along the way, this happened:</p><ul><li>A ServiceNow ticket in plain English, with four people blaming four different things, had to be read by an LLM and still be trusted. Every device the model extracts has to appear in the ticket’s own text, or it’s thrown away</li><li>My first full run burned five investigation rounds and 142,385 tokens and found nothing. The LLM was not the problem. Every hypothesis generated blamed a laptop because that was the only device the incident report named</li><li>The system named the wrong root cause because of a 0.04 gap between two confidence scores. The two hypotheses were not rivals at all: one was a trigger, and the other was the cause. Telling them apart took various modifications: a new step and a code rule that checks the LLM’s answer</li><li>Splitting an overloaded evidence planner agent into two agents did not remove the need for a guardrail</li><li>On the final code version, three runs out of three found the right cause in one or two rounds, for about half the token cost of the failed run. Three runs of one scenario with one model is encouraging, but hardly bulletproof</li></ul><p>If you want the ugly, messy details of everything that broke, keep reading. If you wanted the summary, that’s it.</p><p>This is the eighth piece in the series. <a href="https://medium.com/@icaro_vazquez/agentic-ai-for-the-enterprise-building-an-investigation-team-that-argues-with-itself-7f68fb06d4fc">Article seven</a> ended with a short list of things I haven’t tested yet, and the newly added Scenario Two touches most of them: a source that is not a clean fixture, a more complicated case, and a doubt about our evidence planner agent. I will point them out as they come up.</p><h3>The Ticket that Lies</h3><p>On Monday morning, September 14, 2026, Maria Lopez opens a ServiceNow ticket: Her Microsoft Teams video calls keep breaking up, the video freezes, and her audio is having issues too. She has a theory: her headset is old, IT gave her a new laptop dock last week, and she’s using the VPN; it must be a VPN problem. Her neighbor Ravi at the next desk is having the same issue, but he blames his webcam.</p><p>At a high level, every one of these theories makes sense, and every theory is wrong. Every network link is up; all pings succeed. However, when we look deeper, all these theories are just noise: the headset is a user’s problem (Layer 8 problem in network geek-speak 😁 ), the dock is an IT issue, the VPN client is a network team’s issue, and the webcam is also a Layer 8 problem. Four theories, four different owners, and each one politely saying it’s not their problem. This is the textbook definition of Mean Time to Blame (MTTB) living inside a ServiceNow ticket.</p><p>The real story is in a different ticket: a change record from the night before. At 22:00 UTC, a router upgrade (from firmware version 17.6.4 to 17.9.3) was closed with a note that said all interface configurations were “successfully reapplied from the standard branch template” and that no issues were observed. First of all, kudos to this team: I’m assuming they use config-as-code (the note mentions a branch template), so their router configs live in a repo. However, there was a problem. The interface configs were indeed reapplied successfully, but what was not reapplied was the QoS policy that protects voice traffic on the WAN uplink. Nobody noticed because no traffic needed the policy until another person was pushing a big backup from his workstation. That backup saturated the 100Mbps uplink, hitting 97% utilization. With no policy to protect them, Teams’ packets were dropped along with everything else. This is illustrated in Figure 1 below.</p><p>So, in this scenario, what is the root cause? The backup? Or the missing policy? Hold that thought. It is precisely this question that nearly broke my Agentic AI team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WMpPViv21DgNUP1xMHn10g.png" /><figcaption><strong>Figure 1. Issue Timeline</strong></figcaption></figure><h3>The Agentic AI Team, and what changed</h3><p>You will hear these names in the rest of the article, so here’s the cast, shown in Figure 2 below. The investigation team is a pipeline of specialized agents, each one doing its job, with ordinary code between them.</p><ul><li><strong>The Incident Framing Agent</strong> reads a ticket and works out what is affected and where to look</li><li><strong>The Hypothesis Generator Agent</strong> proposes the possible causes. It runs once per investigation.</li><li><strong>The Evidence Gap Planner</strong> looks at the open hypotheses and decides what evidence is missing</li><li><strong>The Evidence Test Creator</strong> turns each gap into a concrete test the system can run</li><li><strong>Execution</strong> is plain code, not an LLM. It runs tests against data sources and returns observations</li><li><strong>The Evidence Analyst Agent</strong> reads the test results and says which hypotheses they support or weaken</li><li><strong>The Hypothesis Challenger Agent</strong> argues against the leading theory and says what would have to be true for it to be wrong</li><li><strong>Root Cause &amp; Remediation</strong> writes the final recommendation. It only recommends; it never changes anything.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LFcuLHxHm0l7WY6T1PpLpQ.png" /><figcaption><strong>Figure 2. New Agentic AI Investigation Team Architecture</strong></figcaption></figure><p>The planner, test creator, execution, analyst, and challenger form a loop that runs up to five times. A gate written in code, not by a model, decides when to stop: a hypothesis is supported at a 0.80 confidence score or higher, and the Challenger upholds it; the leader stays the same for two rounds; a round returns no new evidence; or the round limit is reached.</p><p>Scenario Two changed this in five ways, and the rest of the article is about them:</p><ol><li><strong>More sources</strong>: ServiceNow tickets and Microsoft Teams call records now feed the evidence, joined by an entity resolver</li><li><strong>A wider view:</strong> The agents are handed the network around the affected device, plus a list of recent changes</li><li><strong>A new step, Causal Resolution</strong>, between the loop and the recommendation, which decides whether two supported hypotheses are rivals, or they complement each other</li><li><strong>A Hypothesis Challenger that must name its rival</strong>, so a dispute can be routed instead of repeated</li><li><strong>A planner that is two agents</strong>, whereas before we used to have only one.</li></ol><h3>The First Run failed, and it was not the Agents</h3><p>My first full run of this scenario burned 5 rounds, 24 LLM calls &amp; 142,385 tokens, and ended with no root cause at all.</p><p>It is easy, and almost an automatic reaction, to first blame the LLM. The LLM was fine, though. Every one of its hypotheses blamed Maria’s workstation, ws1, because ws1 was the only device the context (the tickets) mentioned. The incident report named one endpoint; the hypothesis generator was given that endpoint; and an LLM did what an LLM does when handed a single data point: it described it in detail. The switch, the router, the WAN uplink, and the workstation’s backup were all in the evidence graph. The model never saw any of them.</p><p>The fix was not a better prompt. It was a small piece of deterministic code that builds a <em>network_scope</em> for the agents:</p><ul><li>the neighborhood around the affected device (three hops, at most 40 entities, nearest first)</li><li>the links between them with their capacities and</li><li>the path from the device to every service it depends on</li></ul><p>Same model, one change to what it was allowed to see (plus a line in the prompt pointing to it), and the hypotheses started naming the router and the uplink.</p><p>That was the first sign of the thesis of this article. When an agentic system fails, look at what the agents were handed before looking at the agents. In other words, don’t automatically blame the agents and, by extension, the LLM they use.</p><h3>A Source That Isn’t Tidy</h3><p>Until now, we have tested the architecture on one source, a PagerDuty-shaped entity. The second scenario adds two completely different sources:</p><ul><li>ServiceNow tickets, which are free text, and</li><li>Microsoft Teams call records, which identify the same device in four different ways</li></ul><p>Together with the topology and the operational state, that makes our four sources.</p><p>None of them could agree on what a device is called. The topology calls it <em>ws1</em>. Maria’s ticket calls it <em>WS-0142</em>. A Teams record identifies a device by its FQDN, user principal name, MAC address, or asset tag, depending on which record you read. And Maria’s ticket places Ravi “at the next desk”, which is a location, not an ID. If the agents have to figure out that these are all referring to one machine, they will do it most of the time, and “most of the time” is a bad property in a system whose job is to be trusted.</p><p>Two code snippets handle this. In both cases, I didn’t want the LLM to decide which device is which. A wrong guess would look like evidence, so the rules live in code:</p><ul><li><strong>An entity resolver that never guesses</strong>: It matches identifiers by exact id, MAC address, hostname, or FQDN, using aliases the topology source declares. An identifier that has no matches, or matches two things, comes back as unresolved or ambiguous, and the adapter records a warning. The run prints one every time a Teams call comes from a device nobody has heard of: UNKNOWN-DEVICE-77.corp.example.com does not match any known entity; record skipped. Loud and boring beats quiet and wrong.</li><li><strong>LLM extraction under a microscope</strong>: Tickets are free text, so a model reads them. But every device it extracts must appear in the text of the ticket, or it’s discarded. It must then resolve to a known entity, or it is reported; the ticket reporter’s own theories (the headset, the dock, the VPN client) are stored as unverified theories, never as facts; and malformed output raises an error instead of continuing.</li></ul><p>The pattern generalized for these two sources. I still need to add a Datadog-style feed in a future iteration. Notice all evidence sources have an adapter, and there is now an entity resolver that does the entity matching before handing out all the info to the evidence graph (see Figure 2 above). The agents are in the investigation loop, and the last two steps are new.</p><h3>The Nastier Case: Trigger vs Root Cause</h3><p><a href="https://medium.com/@icaro_vazquez/agentic-ai-for-the-enterprise-building-an-investigation-team-that-argues-with-itself-7f68fb06d4fc">The previous article</a> worried about harder cases, such as three plausible causes at once. Scenario Two has two, and that was enough to break things.</p><p>After adding the new scope, the system found both explanations:</p><ul><li>Hypothesis one: the WAN link is saturated by a backup transfer</li><li>Hypothesis two: the QoS policy is missing from router 1’s WAN interface</li></ul><p>The evidence supported both. In one run, the congestion hypothesis scored 0.82, and the missing policy hypothesis scored 0.78. My pipeline picked the highest-scoring hypothesis as the root cause, so it told the on-call engineer the problem was the backup. A 0.04 gap, on numbers clustered around 0.92, 0.88, 0.82, and 0.78, decided the answer. The engineer would have gone on to throttle the backup, and the network would still be broken.</p><p>The mistake I made was in the shape of the pipeline. It treated hypotheses as rivals, where one wins and the others lose. These two were not rivals. They were linked: the QoS policy was missing (a deviation from how the network was supposed to behave), and the backup was the load that exposed it. I show this dichotomy in Figure 3 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4eUdsSGCgHP72e_z9JLoDQ.png" /><figcaption><strong>Figure 3. Are hypotheses rivals or are they linked?</strong></figcaption></figure><p>The fix is to add a step that sits between the investigation loop and the final recommendation, called causal resolution. When more than one hypothesis is supported, it asks a question: are these hypotheses rivals, or do they complement each other? It also gives the model one test for finding the underlying cause:</p><p>The underlying cause is a deviation from the intended state of the system. A trigger is a normal load the system is designed to tolerate</p><p>The model answers, but code checks the answer. This is another example of why my architecture is divided into agents and executors: The agents reason, but ordinary code checks their output. Nothing in the system actually executes the actions or recommendations put forward by the agents. That stays with a human.</p><p>Every hypothesis ID has to be one of the supported candidates. Calling them linked needs an underlying cause, at least one contributing factor, and a rationale. Anything malformed falls back to treating the hypotheses as rivals and prints why. The final report then names the root cause and lists the trigger as a contributing factor, and it orders a remediation so the policy is fixed before anyone touches the backup.</p><p>The proof came in a later run. There, the trigger had the higher confidence score (0.85 against 0.82), and the system still named the missing policy as the root cause. That is the case the old rule got wrong.</p><h3>The fixes that made it boring</h3><p>The rest of the work was less dramatic, but I think it is the part that matters most in practice. Each fix is a failure I watched happen in a real run:</p><ul><li><strong>A round that learns nothing ends the loop</strong>. The hypothesis challenger kept asking for evidence no source provides, like a pre-upgrade configuration snapshot and the exact time the backup started. The loop planned the same tests again. Now the orchestrator compares evidence ids between rounds, and if a round finds nothing new, it stops and says so</li><li><strong>The Hypothesis Challenger has to name its rival</strong>. A dispute now carries the ID of the hypothesis it believes in instead. If that rival is itself supported, the question is how the two relate, which another round of evidence cannot settle, so the dispute goes to causal resolution. That ended the flip-flopping disputes that used to keep the loop running</li><li><strong>The Generator sees what changed</strong>. In one run, no missing-policy hypothesis was supported, and the system confidently named the trigger as the root cause. The change record was in the evidence graph the whole time, and nobody had seen it. The hypothesis generator and the incident framing agents now get a list of recent changes on the scoped devices, newest first, with a rule that a change is a lead and not a verdict. After that, the policy hypothesis appeared in all three runs</li><li><strong>Even the scorecard had a bug</strong>. My top-1 metric said “wrong” on a correct diagnosis because the hypothesis said “QoS policy regression” and my keyword list said “QoS policy misconfiguration”. I patched the list twice, then stopped: the match is now structural, a QoS mention close to a defect word, in either order. It is still a keyword heuristic, and the output says so.</li></ul><p>None of this makes the LLM smarter. Each fix removes a place where the system relied on the model to be sensible.</p><h3>Did splitting the planner help?</h3><p><a href="https://medium.com/@icaro_vazquez/agentic-ai-for-the-enterprise-building-an-investigation-team-that-argues-with-itself-7f68fb06d4fc">My previous article</a> mentioned I suspected the planner was doing too much in one call: deciding which evidence gaps mattered and also designing the tests to close them. So I split it into an Evidence Gap Planner and an Evidence Test Creator, and kept the guardrail that caps how many gaps a round might pursue.</p><p>The guardrail still fires. The Scenario Two logs still contain lines like “proposed 3 gaps, capping to 2”, so the agent that decides what matters is still asking for more than the loop will pay for. I cannot tell you the split improved results, because I did not run a controlled comparison: the scenario, the context, and the other fixes all changed in the same period. What I can say is that the split made the two jobs easier to read and debug, and that the guardrail is not retired. Part of my suspicion was wrong, or at least not proven.</p><h3>The numbers, honestly</h3><p>On the final code, Scenario Two was diagnosed correctly in three runs out of three, in one or two rounds, using between 53,000 and 88,000 tokens. The runs’ summary is shown in Table 1 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FphSyVkukM8_NsbrDWp5WQ.png" /><figcaption><strong>Table 1. Token Costs for different Agentic AI Investigation Team Runs</strong></figcaption></figure><p>The last three runs averaged about 65,000 tokens, roughly half of the failed run. Figure 4 below shows the token cost for all runs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BohZMckrhRnH3-slgeyuKQ.png" /><figcaption><strong>Figure 4. Token Cost for all runs of the Agentic AI Investigation Team</strong></figcaption></figure><p>A few honest caveats, because a table like this is easy to oversell:</p><ul><li>It is three runs of one scenario with one model. My claim is that it held in three runs, not that it is reliable</li><li>The 142,000-token run is a single failed run, not an average, so it shows what the loop costs when it could not converge</li><li>The round count changes. Run 3 needed a second round because neither hypothesis cleared the 0.80 confidence bar in round one, and its root cause is reported at 0.78. By round two, the leading hypothesis cleared the bar, and the underlying cause is reported with its own confidence</li><li>Scenario 1, a simple BGP link failure, which I measured earlier and did not rerun in the final code, was five of five correct, with one-round runs near 36,000 tokens and two-round runs near 60,000 tokens</li></ul><h3>What is still imperfect</h3><p>The system works on the scenarios I built it for. Here’s what I would not trust yet, starting with what is still untested:</p><ul><li>Only t<strong>wo new sources, and no monitoring feed yet</strong>. ServiceNow &amp; Microsoft Teams generalized the pattern, but I still don’t have a data feed with real metric time series. I’m expecting that adding such a feed will cause a lot of confusion for my Agentic AI team</li><li><strong>Two causes, not three</strong>. Scenario Two has two possible causes. I haven’t tested with three possible causes or with two independent faults at once.</li><li><strong>Two scenarios, selected by me</strong>. We doubled the number of scenarios, but I selected them so the system is still graded on the stuff I picked.</li></ul><p>And the smaller problems I found while doing it:</p><ul><li><strong>The evidence is synthetic</strong>. The scenarios I wrote are realistic but use synthetic data: raw ticket text, call records, and device state. They test reasoning and the contract between the agents. They do not test a real ServiceNow ticket or a real router</li><li><strong>Confidence is rough</strong>. Confidence scores cluster around a handful of values, so a gap of a few hundredths of a point means little. Causal resolution reduces the damage, but the number is still not a measurement</li><li><strong>Concerns written before the verdict can contradict it.</strong> After causal resolution settles a case, I move the settled disputes into their own section and de-duplicate the evidence gaps. The Challenger’s free-text systemic concerns remain, labelled as raised before the resolution, because filtering free text in code would be guesswork and filtering with another model would break my own rule.</li><li><strong>The Hypothesis Challenger can ask for evidence that isn’t there</strong>. When the Challenger disputes a hypothesis, it can only ask for measurements the system knows how to take, like link utilization or packet loss. Code checks that the measurement exists. It does not check that a source has that measurement for this device at that time.</li><li><strong>Analyst rules are warnings</strong>. The Evidence Analyst is told not to write words like ‘confirmed’ or ‘root cause’. When it does, the code prints a warning and carries on</li><li><strong>The scorecard is a rough check</strong>. It asks three things: is the right cause the system’s answer (top-1)?, is it among the top 3 causes (top-3), and did the system point at the right device. The device check is exact. The other two work by keyword matching, so a correct diagnosis worded in an unexpected way can be marked wrong, which happened twice.</li><li><strong>One model, three runs</strong>. Nothing here says how the system behaves with another model, and I have not tested that yet</li></ul><h3>What’s Next</h3><p><a href="https://medium.com/@icaro_vazquez/agentic-ai-for-the-enterprise-building-an-investigation-team-that-argues-with-itself-7f68fb06d4fc">The previous article</a>, article seven in the series, claimed the design was vendor-neutral: a common schema every source speaks, a resolver that refuses to guess, and a registry of what can be measured. Scenario Two is the first real test of this claim, and it held for the two new sources I tried. It also found things the claim could not have predicted: a device that has four different names, a confidence score gap of 0.04 that decided a diagnosis, and an agent that could see Maria’s workstation but not the router behind it.</p><p>There was a failure pattern worth mentioning. Whenever the system went wrong, the fix was a rule in code or a change to what the agents were shown, never a request that the model try harder. This is notable because one of the first ‘fixes’ people try is to change the model prompt to see if the model comes up with a better response. I tried that, and I can tell you it is a time sinkhole: you will spend hours/days trying to find the ‘perfect’ prompt, and the non-deterministic nature of the LLM will still mean you will get unexpected errors. Better to enforce the rule in code or change the data given to the agents.</p><p>I still have one more item to fulfill. Article six ended by asking whether a smaller or an open-source model could run parts of or the whole Agentic AI team. I still have not tested with other LLMs, and this system is a better place to do it than the old one: it has a scorecard, a token count, and two scenarios. That, or a harder scenario with a real monitoring feed, is where I would go next.</p><h3>The series so far</h3><ul><li><a href="https://medium.com/@icaro_vazquez/agentic-ai-a-quick-overview-an-example-3aa538926e53">Agentic AI: A Quick Overview &amp; an Example</a></li><li><a href="https://medium.com/@icaro_vazquez/agentic-ai-a-more-generic-and-more-useful-example-9140b73027d4">Building an Agentic AI Research Team</a></li><li><a href="https://medium.com/@icaro_vazquez/what-observability-revealed-about-my-agentic-ai-team-15e442e6a6d7">What Observability Revealed About My Agentic AI Team</a></li><li><a href="https://medium.com/@icaro_vazquez/building-an-agentic-ai-data-science-team-with-llms-a-multi-agent-machine-learning-workflow-2c6d38ed0f9b">Building an Agentic AI Data Science Team with LLMs: A Multi-Agent Machine Learning Workflow</a></li><li><a href="https://medium.com/@icaro_vazquez/generalizing-an-agentic-ai-data-science-team-lessons-from-three-kaggle-competitions-4107bc75f5cb">Generalizing an Agentic AI Data Science Team</a></li><li><a href="https://medium.com/@icaro_vazquez/agentic-ai-for-the-enterprise-building-an-investigation-team-that-argues-with-itself-7f68fb06d4fc">Agentic AI for the Enterprise: Building an Investigation Team That Argues with Itself</a></li><li>This piece</li></ul><p>Code: <a href="https://github.com/icarovazquez/agentic-ai-investigation-team">github.com/icarovazquez/agentic-ai-investigation-team</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7322e5f933c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI for the Enterprise: Building an Investigation Team That Argues With Itself]]></title>
            <link>https://medium.com/analytics-vidhya/agentic-ai-for-the-enterprise-building-an-investigation-team-that-argues-with-itself-7f68fb06d4fc?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f68fb06d4fc</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Fri, 25 Sep 2026 07:40:46 GMT</pubDate>
            <atom:updated>2026-09-30T07:06:58.003Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q-sWhXBqXr3ZYIbFiylXZQ.png" /></figure><h3>TL;DR</h3><p>I built another iteration of my agentic AI team; this one investigates enterprise incidents—not just networks, but evidence pulled from SaaS tools (PagerDuty, and soon others) most companies already pay for. Along the way:</p><ul><li>An agent “investigated” a case with zero evidence and wrote a fluent, confident, completely wrong answer. Nothing in the system noticed.</li><li>A confidence score meant the opposite of what it looked like, in two different runs, for the same reason.</li><li>I built an agent, a challenger, whose entire job is to argue with the rest of the team — and the first time it actually won, it caught something a real on-call engineer could easily have missed too.</li><li>A vendor’s alert loaded perfectly into the system and never once got tested — because it turned out to already be a known fact, not a hypothesis.</li><li>Building this yourself, instead of buying the SaaS version, might be the better long-term bet —no vendor can ever be neutral about which tools it treats as equally trustworthy, but you can. Commoditizing SaaS Vendors by making them just another data source, i.e., dumb pipes like ISPs, is just a bonus.</li></ul><p>If you want the mechanics, the code, and everything that broke along the way — keep reading. If you just wanted the quick summary, that’s it.</p><p>This is the seventh piece in a series I’ve been writing about building Agentic AI systems, and it follows a pattern the last two already set. <a href="https://medium.com/@icaro_vazquez/building-an-agentic-ai-data-science-team-with-llms-a-multi-agent-machine-learning-workflow-2c6d38ed0f9b">Article five</a> took the multi-agent team I’d originally built for corporate strategy research and turned it into something that could compete in Kaggle competitions. <a href="https://medium.com/@icaro_vazquez/generalizing-an-agentic-ai-data-science-team-lessons-from-three-kaggle-competitions-4107bc75f5cb">Article six</a> pushed that further — generalizing the same architecture across three different competitions, to see what actually held up versus what was Titanic-shaped luck.</p><p>This piece is the same move, a third time, in a domain I haven’t written about yet: I took that same underlying architecture — agents that reason, deterministic code that fetches evidence, a strict contract between the two — and rebuilt it for network incident investigation. Then, over the past several weeks, I pushed it again, from “investigates a network” to “investigates an enterprise,” pulling in evidence from the SaaS vendor tools that already sit in most companies’ stacks. I feel that “investigating an enterprise” is a more interesting read than “investigating a network”. When an enterprise has a problem, it doesn’t care where it’s coming from; it just wants it fixed. The enterprise’s organization (different teams focused on different domains) is what makes teams concentrate on or investigate one area; i.e., the NOC will check the network for problems and, if it’s not the network, they are out of the loop, and another team has to take the baton to the conclusion. In my experience working in and selling to enterprises, Mean Time To Blame (MTTB) is a real thing that happens everywhere. Organizational silos create hand-offs. Hand-offs create finger-pointing. Finger-pointing is the actual disease this architecture is meant to cure.</p><p>If you’ve read the earlier pieces, the shape will be familiar: I’ll show you what broke, what that taught me, and what I built to fix it. If this is the first one you’re reading, here’s the one-paragraph version of everything before this point: Agentic AI systems fail in specific, learnable ways — not “the model is dumb,” but things like agents drifting from their contracts under pressure, evidence that silently never reaches the code that’s supposed to analyze it, and confidence scores that mean the opposite of what you’d assume. Six articles in, I’ve gotten reasonably good at catching these before they become a bad answer nobody questioned. This piece is what that looked like applied somewhere new.</p><h3>Investigating Issues in an Enterprise</h3><p>Every enterprise I’ve worked with or talked to runs the same play: buy the observability platform, buy the SIEM, buy the ticketing system, buy three more tools that promise to catch the thing the last three missed. And every one of them will, eventually, tell you something is wrong. A threshold crossed. An anomaly detected. A synthetic check that failed at 2:30 am.</p><p>What none of them will tell you is <em>why</em>.</p><p>That part still falls on a person. A network engineer opens the alert, opens four other dashboards in four other tabs, and starts doing the actual work: forming a hypothesis, going and checking it, revising when it’s wrong, checking again. The vendor’s dashboard was never wrong, exactly — packet loss really was 100% on that link — it just never had an opinion about what to do with that fact. That’s not a limitation anyone’s hiding. It’s the business model. A monitoring vendor’s job is to watch <em>their</em> slice of your infrastructure and tell you when it looks off. Investigating why and deciding what to do about it was never part of the product.</p><p>I’ve spent the last several months building the thing that’s supposed to sit in that gap: an Agentic AI team that takes in exactly the kind of evidence those vendor tools already produce — and instead of stopping at “here’s an alert,” does the actual investigation. Forms hypotheses. Goes and gets evidence. Argues with itself before it commits to an answer. Writes up a root cause and a remediation plan a human can act on.</p><p>This isn’t a new idea — and I’ll get to that in a minute, because it matters. What I think is actually worth reading isn’t the pitch. It’s the build log: the specific ways this kind of system breaks, quietly, in ways that look fine until you check closely — and what it took to make it trustworthy enough to hand someone a real answer.</p><h3>Is This Unique?</h3><p>Before I go further, it’s worth being honest about something: this isn’t uncontested ground anymore. Over the past year, AWS shipped a DevOps agent that does a version of this. New Relic has an SRE agent that claims a real chunk of resolution-time savings. Instana, ITRS, and a handful of smaller players are all running variations of the same pitch — an agent that correlates your telemetry, forms a hypothesis, and tells you what’s wrong in plain English. If you searched for “AI agent root cause analysis” six months ago, you’d have found a handful of startups. Search it today, and you’ll find a market.</p><p>I’m not going to pretend I got there first, because I didn’t, and I’m not going to pretend this is a finished product, because it isn’t — it’s a Colab notebook I keep breaking and fixing. What I can offer that a vendor’s launch blog can’t is the part product announcements usually leave out: what it actually took, mechanically, to get an LLM-based investigation to a point where I’d trust its answer. Not the demo. The failure modes. The bug where evidence loaded into the system and then simply vanished before anything downstream could see it. The one where a confidence score meant the exact opposite of what it looked like it meant, and the system didn’t tell me — I had to catch it by reading closely. The agent I built specifically to argue with the rest of the team, and what happened the first time it actually won an argument.</p><p>That’s the difference I think is worth writing about. Not “here’s a system that finds root causes.” Here’s what it takes to build one you can actually believe.</p><h3>Wait, Don’t the Vendors Already Do This?</h3><p>Here’s the part I don’t think any of those vendors will ever say in their own marketing: their agents aren’t worse than what I’ve built. They’ve probably got the edge, honestly — real engineering teams, thousands of customers’ worth of pattern-matching, and a compute budget that would make my Colab notebook blush. And they’re not standing still, either — Datadog can pull in a PagerDuty incident now too; plenty of platforms are adding connectors. But when Datadog ingests a PagerDuty alert, it’s still Datadog deciding what that alert means, still Datadog’s own model of your infrastructure the alert gets mapped onto, still Datadog’s product roadmap deciding which vendors are worth building a connector for next. A connector isn’t neutrality. The enterprise is the only party that has no product to protect by staying inside its own walls — the only one that can decide every system, including the observability vendor itself, is just one more source of evidence, on equal footing with everything else.</p><p>The enterprise is the only party sitting at the vantage point that spans all of it. That’s not a nice-to-have. It’s the whole game.</p><p>I don’t think this makes every SaaS observability tool a bad purchase, or means you should go rip out your monitoring stack out of spite. It means something narrower, and more useful: once you have an architecture like the one I’m about to walk through — where every source, whatever it is, gets normalized into one evidence graph through its own small adapter — every vendor you’re already paying starts to look less like a platform you depend on and more like a data source you happen to be buying from one of several possible providers. Datadog’s agent stops being the thing doing your reasoning and becomes one more feed into the thing that is.</p><p>That’s a real shift in leverage, and no vendor is going to hand it to you out of the goodness of their product roadmap. In a sense, we are turning them into the SaaS version of ISPs: dumb pipes, but in this case, they just become a data source. From the reasoning layer’s point of view, that makes the vendor increasingly interchangeable and a bit of a commodity: its differentiation remains valuable inside its platform, but upstream it becomes just one more data source behind an adapter. The real breakthrough — discovering the root cause and remediating it — is handled by this Agentic AI team, not the vendor.</p><p>I do want to be honest about the other side of this, because it’s real: none of it is free. Vendors have dedicated ML teams and patterns learned across thousands of customers that I will never have. I’m one guy with a notebook and an unreasonable number of “wait, why did the confidence score do that” moments at odd hours. And vendors won’t make this easy on purpose — clean, well-documented API access sometimes costs extra, sits behind an enterprise tier, or just isn’t the well-trodden path their own product team bothered to pave. I hit a miniature version of this friction myself, just getting a PagerDuty-shaped adapter working cleanly. The real version — enterprise scale, a dozen vendors, none of them talking to each other — is a lot harder than the toy version I built.</p><p>So call it a long-term investment, not a free lunch: the upfront cost is real, and the payoff is the one thing no vendor is ever going to be able to sell you, because selling it to you is the opposite of their business model.</p><p>To show why, it helps to look at what a typical vendor pipeline actually looks like on the inside — not any specific company’s, but a fairly representative shape most SaaS platforms share. Figure 1 below shows such an architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GnyLCIACyg_FI0uk_QXUoA.png" /><figcaption><strong>Figure 1. Generic SaaS Vendor Architecture</strong></figcaption></figure><p>Everything in the top layer is shared, single-vendor plumbing. It forks once into the path that draws your dashboards, and again into the data platform, which itself splits three separate ways: a plain data product with no AI in it at all, a classic MLOps pipeline, and — increasingly, as every vendor rushes to ship one — a GenAI feature layered on top, whether that’s a chatbot or something closer to an actual agent. None of those three paths were ever built with a seam for a competitor’s telemetry to enter — there’s no step in any of them where that would even make sense, given how each is organized. They all converge on one place: the API layer, where the vendor’s data and inference results leave their walls and reach you, no matter how that data was produced upstream. The double-headed arrows in the diagram aren’t decoration — Kafka consumers pull from a topic rather than being pushed to, and the API layer works the same way, calling out to whichever of the three paths a request actually needs and getting a response back. Each of those calls stays entirely inside the vendor’s walls until it reaches you at that one exit point. That convergence is not incidental. That’s exactly the seam an adapter plugs into. <strong>Two things can happen to that response once it clears the API layer: it gets rendered in the vendor’s own UI, or it lands directly in your own systems, if you’re consuming the product purely through its API. That second path is the one that matters here — nothing else in this diagram was built for you to plug into. That one arrow is.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0DGMOwA6ktNYycBMWH_jqA.png" /><figcaption><strong>Figure 1A. The Adapter bridges the SaaS API to our Evidence Graph</strong></figcaption></figure><p>This exact pattern repeats for every vendor — Datadog’s API, ServiceNow’s API, whichever one you’re integrating next. One small adapter each. Nothing else in your architecture ever has to know or care which vendor is on the other end.</p><h3>My Agentic AI Investigation Team Architecture</h3><p>Okay, enough vendor-shaming. Here’s what I actually built, as briefly as I can describe it and show it in Figure 2 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lNiLVtHSfbkxDF1Bt-P3cw.png" /><figcaption><strong>Figure 2. Agentic AI Investigation Team Architecture</strong></figcaption></figure><p>Every investigation starts with an <strong>Evidence Graph</strong> — a normalized model of entities (routers, hosts, services) and whatever’s been observed about them (metrics, events, vendor alerts). The graph doesn’t care about the vendor-specific shape a piece of evidence arrived in, and downstream reasoning shouldn’t have to. That’s the job of an <strong>adapter</strong>: one small, boring piece of code per source, whose entire purpose is translating that source’s native format into the graph’s normalized shape. One adapter for NIKA’s network telemetry. One for a PagerDuty incident. Add a new source, write a new adapter, and nothing downstream even notices.</p><p>On top of the graph sit <strong>deterministic executors</strong> — plain code, no LLM involved, that answer specific, boring, reliable questions: is this entity reachable from that one, what’s the blast radius if this link drops, what does this interface’s state actually say right now. These get wrapped into <strong>capabilities</strong> — network state, reachability, vendor alerts — which is the only thing an agent is ever allowed to touch. No agent gets to see a raw API response, poke at the graph directly, or improvise its own query. It asks a capability a question and gets an answer, same as everyone else.</p><p>The agents themselves only ever see what a capability hands them. The agents are:</p><ul><li>An <strong>Incident Framing</strong> agent turns a vague symptom into a scoped investigation.</li><li>A <strong>Hypothesis Generator</strong> proposes falsifiable explanations — falsifiable being the operative word, since “something’s wrong with the network” isn’t a hypothesis; it’s a shrug.</li><li>An <strong>Evidence Planner</strong> decides what to check and how, deterministically, in code — because I learned the hard way that letting an LLM sort its own priorities is a great way to get a different answer every time you ask.</li><li>An <strong>Evidence Analyst</strong> reads back what was found and scores each hypothesis against it, honestly, ideally without lying about how sure it is.</li><li>And then — this is the part with the most scar tissue, and where I’ll spend most of the rest of this piece — a <strong>Hypothesis Challenger</strong> tries to actively tear the leading answer apart before anyone’s allowed to commit to it.</li><li>A <strong>Root Cause &amp; Remediation</strong> agent only gets to write the final recommendation once that fight is over.</li></ul><p>One thing worth being explicit about, since it matters more here than it would for a Kaggle leaderboard: the Root Cause &amp; Remediation agent writes a <em>recommendation</em>, not an action. Nothing in this system touches a router, a firewall rule, or a config file. No enterprise I’ve ever worked with is going to let an agentic system automatically modify a production network on the strength of an LLM’s confidence score, and none of them should — the whole architecture assumes a human reads the recommendation, the evidence behind it, and the Challenger’s dispute, and decides. That boundary isn’t an afterthought I’m adding for this article; InvestigationPolicy has carried a require_human_approval flag since early in the project, for exactly this reason. I haven&#39;t built the actual approval workflow around it yet — right now it&#39;s a documented intention more than an enforced gate — but the assumption was there from the start: this team investigates and argues its way to an answer, and a person decides what happens next.</p><p>Normalizing every source into one evidence graph doesn’t mean throwing away where each piece came from, either. Every record — an observation, an event, whatever it is — still carries a source_id back to the exact adapter that produced it. So when a recommendation cites &quot;supporting evidence,&quot; that&#39;s not just a sentence the model wrote; it traces back to a specific vendor and a specific payload. That matters for the same reason require_human_approval matters: a person deciding whether to trust a recommendation should be able to ask &quot;wait, which system actually told us this,&quot; and get a real, checkable answer — not just the agent&#39;s word for it.</p><p>(The actual code — adapters, capabilities, agents, and all — is <a href="https://github.com/icarovazquez/agentic-ai-investigation-team">on GitHub</a>, if you want to see the real thing instead of my description of it.)</p><p>That’s the whole shape. None of it is exotic. One thing the diagram doesn’t show: steps three through five aren’t strictly linear. If the Challenger disputes the leading hypothesis, the team goes back and plans another round of evidence — up to a limit — before anyone’s allowed to write a final answer. That loop, and everything that went wrong building it, is most of what the rest of this piece is actually about.</p><p>Right now the loop doesn’t run forever. It stops when the Challenger accepts the leading hypothesis, when no useful discriminating evidence remains, or when the investigation hits its configured round limit. That last case is important: hitting the limit doesn’t magically turn uncertainty into certainty. The correct output then is “this is the leading explanation and here is what remains unresolved,” not a manufactured definitive root cause.</p><p>None of that shape was obvious on day one, though. I built it the slow way — write something, watch it produce a confident wrong answer, figure out why, fix it, repeat — and most of what’s actually worth reading here isn’t the architecture. It’s the specific, repeatable ways this kind of system fails, and what each failure taught me to build in response. Four of those stood out enough to write down.</p><h3>Hard-Learned Lessons</h3><p><strong>The evidence that was never there</strong></p><p>The first real bug wasn’t in an agent. It was in a config file, and it didn’t throw an error.</p><p>I’d built two adapters for the network scenario I was testing against: one that loaded the topology (which routers connect to which), and one that loaded the operational telemetry — interface status, BGP session state, the actual signal that something was broken. I wired up the investigation, ran it end to end, and it worked. Hypotheses got generated. Evidence got “collected.” A root cause got written up.</p><p>It was <em>wrong</em>. Not because the reasoning was bad — because there was no evidence for it to reason about. The config I’d written only listed the topology source, as shown in the code snippet below. The operational adapter existed, was fully coded, was registered correctly — and was never once referenced by the one config object that actually decides what gets loaded. The evidence graph loaded four entities and three relationships and exactly zero observations, every single run, and nothing in the system so much as blinked. Every agent downstream just did its job with an empty hand and produced a confident-sounding answer anyway.</p><pre>evidence_sources=[<br>    EvidenceSourceConfig(<br>        source_id=&quot;nika_simple_bgp_topology&quot;,<br>        adapter_name=&quot;NIKASimpleBGPAdapter&quot;,<br>        ...<br>    ),<br>    # This second source was never here. Nothing raised an error<br>    # about its absence — the graph just quietly had no observations.<br>],</pre><p>That’s the scary part. Not that the bug existed — bugs exist — but that nothing about the output looked broken. The hypotheses were plausible. The prose was fluent. If I hadn’t happened to print the evidence graph’s summary and noticed observations: 0 sitting there, I would have shipped an investigation team that was, functionally, making things up with excellent grammar and was full of confidence in itself.</p><p>The fix was one line — add the missing source to the config. The lesson took longer to sink in: an agentic system’s biggest risk isn’t that it gets something wrong. It’s that it gets something wrong <em>fluently</em>, with no signal anywhere that it was working from nothing. Every fix I made after this one, for the rest of the project, was in some way downstream of that lesson — build the thing that would have caught this, not just the thing that fixes this specific instance of it.</p><p><strong>The Confidence Score that lied</strong></p><p>The second one was subtler, and it took longer to notice because the system was <em>right</em> every time it happened — right for the wrong reason, which is a worse kind of wrong.</p><p>Every hypothesis my Evidence Analyst agent scores gets a status — supported, rejected, weakened — and a confidence number attached to it. The idea is simple: if a hypothesis is rejected, that confidence should trend toward zero, because you’re not confident it’s the cause. I wrote that into the prompt, in plain English, more than once.</p><p>Then I ran the same investigation twice, back-to-back, same config, same evidence. Once, a rejected hypothesis came back with confidence 0.05. The next run, an equally rejected hypothesis — same status, same underlying evidence — came back with confidence 0.95, as shown in Figure 3 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FQifaolubD17_7skKxNBfg.png" /><figcaption><strong>Figure 3. Confusing LLM output</strong></figcaption></figure><p>The model wasn’t malfunctioning. It had just quietly picked a different convention each time: sometimes “confidence this is the cause,” sometimes “confidence in my own rejection of it.” Both are defensible readings of the word “confidence.” Neither was consistent, and nothing forced it to be.</p><p>This one mattered more than it might sound, because I was about to build the part of the system that picks a winner — the piece that looks at every hypothesis’s confidence and decides which one gets written up as the answer. If that piece trusted the raw number, a firmly, correctly rejected hypothesis at 0.95 could out-rank a genuinely supported one sitting at 0.85, purely because the model had picked the other reading of “confidence” that run. A coin flip embedded three layers deep in a system that was supposed to be making a defensible decision.</p><p>The fix wasn’t a better prompt. I’d already tried that, and it held for a while and then quietly stopped working, the way prompt-only fixes tend to. The real fix was to stop trusting the model’s number at all for this one specific thing: any hypothesis marked “rejected” gets its confidence floored to zero in code (relevant snippet shown below), no matter what number the model attached to it. Not “trust but verify.” Just — don’t ask the question. A rejected hypothesis is not a candidate, full stop, regardless of what number is sitting next to it.</p><pre>def directional_confidence(assessment: HypothesisAssessment) -&gt; float:<br>    &quot;&quot;&quot;<br>    &quot;How likely is this hypothesis to be the true cause&quot; —<br>    normalized regardless of which convention the model happened<br>    to use for &quot;confidence&quot; this particular run.<br>    &quot;&quot;&quot;<br>    if assessment.status == HypothesisStatus.REJECTED:<br>        return 0.0<br>    return assessment.confidence</pre><p>That’s become a general rule for me now, not just a one-off patch: anywhere a wrong answer would silently corrupt something downstream, don’t ask an LLM to police itself. Write code that makes the wrong answer structurally impossible to act on, and save the model’s judgment for the places where being wrong just produces a bad-but-visible answer instead of a quiet, load-bearing lie.</p><p><strong>The agent who loved to argue back</strong></p><p>By the time I built the third piece, I’d learned not to fully trust a single agent’s judgment, even a well-scored one. So I built a fourth agent whose entire job was to disagree.</p><p>The Hypothesis Challenger doesn’t collect new evidence. It gets handed the leading hypothesis — the one currently winning — and is explicitly instructed to argue against it, using nothing but the evidence already on the table. Find the hole. If there isn’t one, say so. If there is, say what additional evidence would close it. It’s the one agent in the whole system whose job is to be annoying on purpose. Any reader who is a contrarian should be jumping for joy now: I added one of your own to the team.</p><p>The first few times I ran it, it upheld everything. The leading hypothesis was well-supported; the Challenger looked, found nothing, said so, and the investigation moved on. I remember being a little disappointed — I’d built an adversary, and it kept agreeing with everyone.</p><p>Then I ran it against a case where the leading hypothesis was “physical link failure,” sitting at 0.9 confidence, built on genuinely strong evidence: both interfaces on both routers reporting down, at the same time, in both directions. The Evidence Analyst’s own rationale said the symmetry “rules out unilateral router failures.” That sounds right. It’s also not quite true, and the Challenger caught it: a single router’s interface failing on its own side would <em>also</em> produce a symmetric-looking down state on both ends of the link, because the link itself would be non-functional from both directions regardless of which side actually broke. The evidence didn’t distinguish “the link is physically severed” from “one router’s port died and took the whole link down with it.” Nothing in the investigation had actually tested for that difference. It had just sounded thorough.</p><p>That’s a genuinely good catch, and I didn’t teach it that. I gave it the evidence and an instruction to be skeptical, and it found a real gap in reasoning a human on-call engineer could easily have missed at 2 am too. The recommendation that shipped afterward didn’t pretend the dispute never happened, either — I’d built the Root Cause agent to require that it acknowledge the Challenger’s specific objection, not just note somewhere that a review had occurred. So the final write-up said, plainly: here’s the leading explanation, here’s the counter-argument that couldn’t be fully ruled out, here’s what a real diagnosis would need to check to be sure. Sometimes being a contrarian pays off handsomely.</p><p>I wish I could tell you it was clean after that. It wasn’t. Once the Challenger could send the team back for another round of evidence, as shown in Figure 4 below, I had a real reasoning loop on my hands — and loops expose everything a single pass lets you get away with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tmJaiVXWfMSA6Vix3wRhcA.png" /><figcaption><strong>Figure 4. Reasoning Loop in Action</strong></figcaption></figure><p>The same evidence-planning agent that worked fine once started returning hypothesis IDs in whatever format it felt like that run — sometimes&quot;H1&quot;, sometimes the full ID with a colon and a title attached, once just the bare digit &quot;1&quot;, and eventually, memorably, the hypothesis&#39;s entire title with no ID at all. I fixed each one as it showed up, felt briefly clever, and then watched the next run invent a fifth format I hadn&#39;t guarded against. Eventually I stopped trying to anticipate the next format and just wrote a resolver with a chain of fallbacks: check for an exact match first, then a leading “h” + digit, then any digit anywhere in the string, and if none of that works, compare the returned text against every hypothesis’s actual title and take the one that matches. This resolver is shown in the code snippet below.</p><pre>def normalize_hypothesis_id(returned_id, hypothesis_set):<br>    canonical_ids = {h.hypothesis_id for h in hypothesis_set.hypotheses}<br><br>    if returned_id in canonical_ids:<br>        return returned_id<br><br>    normalized = returned_id.strip().lower()<br><br>    # Match a bare digit anywhere: &quot;H1&quot;, &quot;h1&quot;, &quot;1&quot;, &quot;H1: some title&quot;<br>    match = re.search(r&quot;\d+&quot;, normalized)<br>    if match:<br>        candidate = f&quot;{hypothesis_set.incident_id}-h{match.group(0)}&quot;<br>        if candidate in canonical_ids:<br>            return candidate<br><br>    # Last resort: the model returned the hypothesis&#39;s title, with<br>    # no ID reference at all. Match against the titles we actually have.<br>    for hypothesis in hypothesis_set.hypotheses:<br>        if hypothesis.title.strip().lower() == normalized:<br>            return hypothesis.hypothesis_id<br><br>    raise ValueError(f&quot;Unresolvable hypothesis_id: {returned_id}&quot;)</pre><p>Ugly, in the way defensive code against an unreliable narrator is always a little ugly. It also hasn’t broken since.</p><p>One round it planned seventeen separate evidence tests when four would have done, and the response got cut off mid-sentence before it even finished being wrong. I spent longer debugging the <em>mechanism that lets agents disagree productively</em> than I spent building the agents themselves. Every fix was another small, specific guardrail — cap the number of tests a planner can request, stop assuming a &quot;fixed&quot; bug stays fixed once you&#39;ve added something new nearby.</p><p>None of that makes it into a vendor’s launch announcement. It’s also, genuinely, the actual work. Building an agent that argues turned out to be the easy part. Building the plumbing so that argument doesn’t quietly collapse the whole investigation was where I spent a lot of my time.</p><p>I think that’s the most honest thing this whole system does, mess included. Most agentic demos show you the confident final answer. This one shows you the argument that almost changed it, and everything it took to make room for that argument to actually happen.</p><p><strong>The vendor alert nobody asked a question about</strong></p><p>This last one isn’t a bug. It’s a finding I didn’t expect, and it’s the one that actually changed how I think about what “using” evidence means.</p><p>Once I had the core network-investigation loop working, I wanted to test the actual premise of this whole piece: could the system take in evidence from an outside SaaS vendor, the way a real enterprise investigation would. I built an adapter that consumes a PagerDuty-shaped incident — same fields a real PagerDuty webhook would carry, deliberately including the one line a real vendor alert always includes and never resolves: <em>no further diagnosis available, see network team for root cause.</em> Talk about being unhelpful; imagine reading that at 2 am when you are on call, and that’s part of what you receive. That sentence is the entire thesis of this article, sitting inside a fixture.</p><p>I wired it in, ran the investigation, and checked whether the evidence-planning agent ever asked for it. It hadn’t. Not once, across several runs. The vendor alert loaded fine, sat in the evidence graph exactly as intended, was fully queryable — and the planner never wrote a single test targeting it. My first read was that I’d built a source nobody could actually reach, and I went looking for the wiring bug.</p><p>There wasn’t one. What was actually happening was more interesting: the vendor’s incident had already been folded into the investigation’s framing, as a known fact, before planning ever started — “vendor alert PD1234567 reported 100% packet loss on the router1-router2 link, triggered at 00:00:05, assigned to NetOps On-Call.” The planner wasn’t ignoring the vendor alert. It had already used it, at an earlier stage, to help write the very question the investigation was trying to answer. There was nothing left to test, because it wasn’t a hypothesis. It was a fact you start with.</p><p>That’s a real distinction, and I don’t think I’d have found it without building the thing and watching it not do what I expected. Evidence isn’t one category. Some of it is a live question with an unknown answer — is this interface up or down right now — and belongs in the loop of things that get planned, tested, and re-tested. Some of it already arrived as a settled fact the moment the phone rang, and its whole value is in framing the investigation correctly from word one, not in being re-verified. A vendor’s alert is almost always the second kind. Building a system that couldn’t tell the difference would have meant either wasting a planning cycle “testing” something already known, or — worse — having the framing agent shrug at what the vendor already told it and start the investigation from a blank page anyway, which is exactly the behavior every enterprise I’ve watched actually complains about.</p><h3><strong>Does It Actually Get It Right?</strong></h3><p>Everything up to this point has been about whether the system behaves the way I intended — evidence loads, agents disagree productively, the loop terminates. None of that answers the harder question I kept coming back to in article four: correct, or just plausible-sounding?</p><p>The NIKA scenario I’ve been testing against comes with a known injected fault — a real answer key, not something I’m grading myself against. So alongside the pipeline itself, there’s a small evaluation function that checks the final recommendation against it: does the winning hypothesis’s cause match what was actually injected, and does it correctly name the entities responsible — not just “something on the network,” but <em>this</em> router and <em>that</em> one.</p><p>On the run I’ve referenced throughout this piece — the one where round one couldn’t distinguish physical failure from a BGP-driven cause, and the system reached for deep diagnostics on its own to resolve it — the check came back clean: correct root cause, and both affected routers named exactly, with nothing extra and nothing missing. That’s one scenario, checked one honest way, and I don’t want to oversell it — it’s not a benchmark, and I only have ground truth because I built the fixture myself. But it’s the difference between “the write-up sounded confident” and “the write-up was actually right,” and after building a system whose first real bug was producing a fluent, wrong answer with no one checking, I wasn’t going to ship a second version that still only checked its own homework by reading its own prose back to itself.</p><p>Correctness isn’t the only thing worth measuring, either. Every LLM call in this system is traced through Langfuse — the same setup I wrote about in <a href="https://medium.com/@icaro_vazquez/what-observability-revealed-about-my-agentic-ai-team-15e442e6a6d7">article three</a> — and it’s just as useful for catching cost as it is for catching bad reasoning. The clearest example: every time the Evidence Analyst’s output needed a repair pass, that one retry cost about as many tokens as an entire extra agent call. A bug that looks free from the outside — “oh, it just retried” — isn’t free. It’s a second LLM invocation you’re paying for, every time.</p><h3><strong>What’s Still Unfinished</strong></h3><p>Before I get to the gaps, it’s worth saying plainly what actually happened here, because it’s easy to undersell after four stories about things breaking: this system took a genuinely ambiguous network failure — one where the strongest evidence pointed at two different root causes depending on how you read a single number — and got to the right answer anyway, with a documented argument for why the runner-up was wrong, not just a confident guess that happened to land. It caught its own silent failures. It disputed itself and came out the other side with a better answer than its first pass. None of that was true six months ago, and none of it happened by accident — every piece of it is a direct answer to a specific way I watched it fail first.</p><p>That said, I meant it literally at the start when I said this isn’t a finished product. A few honest gaps, in the order I actually care about them:</p><p>The reasoning loop works, and it worked well enough in real runs that I stopped needing to fake ambiguity to test it — the planner reaches for deeper diagnostics on its own when the first-pass evidence doesn’t add up, the way an experienced engineer would. But “works” here means “worked on the cases I’ve actually thrown at it,” and the hardest one I’ve run went four rounds before I hit a bug I hadn’t seen yet. I don’t know yet what this loop does on a genuinely nastier case — one with three plausible causes instead of two, or evidence that stays ambiguous even after the deep diagnostics come back. Finding out is next.</p><p>Everything cross-vendor in this piece rests on exactly one real integration: a PagerDuty-shaped fixture. The argument about commoditizing SaaS vendors is only as strong as the adapter pattern actually generalizing past one vendor, and I’ve built the pattern, not the proof. A second, structurally different source — a real Datadog alert, or something messier like unstructured ticket text that needs actual extraction rather than clean field-mapping — is the test that argument still owes the reader.</p><p>And underneath both of those is a question I don’t have a clean answer to yet: how much of what I spent this project fixing was really about agent reliability, versus one specific agent being asked to do too much in a single call — sort priorities, respect a numeric cap, format valid structured output, and reason about prior-round gaps, all at once, every time. Every fix I made was a guardrail wrapped around that agent from the outside. I haven’t yet tried the more radical version of the fix, which is asking it to do less per call and see if the guardrails become unnecessary instead of just effective.</p><p>And the evaluation itself is thinner than I’d like. Right now it’s one scenario, with ground truth I wrote myself, checked one way: does the winning hypothesis match the injected fault, and are the named entities correct? That’s real signal, not nothing — but it doesn’t tell me whether the system stays correct across a range of fault types, whether the Challenger’s dispute rate holds steady or drifts as scenarios get harder, or whether “supported at 0.90” means the same thing across ten different investigations that it meant in this one. A single green checkmark on a scenario I built myself is a start, not a track record. Building out a real evaluation set — multiple fault types, multiple vendors, enough runs to see whether the numbers are stable rather than lucky — is next on the list, right behind the second vendor integration.</p><h3><strong>Where this leaves me</strong></h3><p>Here’s the thing I keep coming back to. Every vendor in the market I named earlier is selling a version of “our agent finds your root cause.” That’s a real product, and for a lot of companies it’s a genuinely good one. But it’s still a product built around its own slice of the problem, with real incentives — a roadmap, a customer base, a thing they’re already selling — to keep it that way even as they add connectors. And the enterprises buying it are the ones organized in a way that makes the actual disease, the hand-off, the finger-pointing, the Mean Time To Blame, worse, not better. No vendor is coming to fix that, because fixing it isn’t their business. It’s yours, if you want it.</p><p>That’s the case for building this instead of buying it: not that it’s cheaper, or faster, or that I did it better than a team with real budget and real customers could. It’s that the one thing worth having — a system with no product to protect, free to treat every tool you pay for, including whichever vendor sold you the best agent, as just one more source of evidence — isn’t something any vendor’s roadmap is built to hand you, connectors or not. The part about commoditizing the SaaS vendors by making them just another data source is just a bonus. I didn’t get here because I’m smarter than the people building the vendor version. I got there because I was willing to build the boring adapter, chase the confidence score that lied, and sit with an agent arguing with itself until it said something true. That part isn’t proprietary. Anyone willing to do the same slow, unglamorous work can get there too.</p><p>The <a href="https://github.com/icarovazquez/agentic-ai-investigation-team/blob/main/notebooks/Agentic%20AI%20Investigation%20Team.ipynb">repo</a> is public, mid-refactor, and exactly as messy as this article describes — if you want to see what “unfinished” actually looks like from the inside, that’s where it lives.</p><p>If you’ve made it this far, you already know what the next piece in this series is going to be about.</p><p><strong>This is part of a series on building agentic AI systems:</strong></p><ol><li><a href="https://medium.com/@icaro_vazquez/agentic-ai-a-quick-overview-an-example-3aa538926e53">Agentic AI: A Quick Overview, An Example</a></li><li><a href="https://medium.com/@icaro_vazquez/agentic-ai-a-more-generic-and-more-useful-example-9140b73027d4">Agentic AI: A More Generic and More Useful Example</a></li><li><a href="https://medium.com/@icaro_vazquez/what-observability-revealed-about-my-agentic-ai-team-15e442e6a6d7">What Observability Revealed About My Agentic AI Team</a></li><li><a href="https://medium.com/@icaro_vazquez/improving-my-agentic-ai-team-how-better-contracts-structured-outputs-and-evaluations-improved-6dc0651cb287">Improving My Agentic AI Team: How Better Contracts, Structured Outputs, and Evaluations Improved…</a></li><li><a href="https://medium.com/@icaro_vazquez/building-an-agentic-ai-data-science-team-with-llms-a-multi-agent-machine-learning-workflow-2c6d38ed0f9b">Building an Agentic AI Data Science Team with LLMs: A Multi-Agent Machine Learning Workflow</a></li><li><a href="https://medium.com/@icaro_vazquez/generalizing-an-agentic-ai-data-science-team-lessons-from-three-kaggle-competitions-4107bc75f5cb">Generalizing an Agentic AI Data Science Team: Lessons from Three Kaggle Competitions</a></li><li>This piece.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f68fb06d4fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/agentic-ai-for-the-enterprise-building-an-investigation-team-that-argues-with-itself-7f68fb06d4fc">Agentic AI for the Enterprise: Building an Investigation Team That Argues With Itself</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generalizing an Agentic AI Data Science Team: Lessons from Three Kaggle Competitions]]></title>
            <link>https://medium.com/@icaro_vazquez/generalizing-an-agentic-ai-data-science-team-lessons-from-three-kaggle-competitions-4107bc75f5cb?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4107bc75f5cb</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 01:57:28 GMT</pubDate>
            <atom:updated>2026-08-01T01:57:28.704Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BUeYImHcChvywbV6N-lWKw.png" /></figure><h3>Generalizing an Agentic AI Data Science Team</h3><p>This article is the next step in a series of Agentic AI articles that aim to introduce and demystify the technology. <a href="https://medium.com/@icaro_vazquez/building-an-agentic-ai-data-science-team-with-llms-a-multi-agent-machine-learning-workflow-2c6d38ed0f9b">In my previous article</a>, I introduced an Agentic AI Data Science Team — a collection of specialized LLM agents and deterministic Python executors that collaborate to solve Kaggle competitions. The architecture itself evolved from an earlier Agentic AI Research Team that I built to automate technical research. The central idea remained the same: let Large Language Models perform reasoning while deterministic Python code performs execution.</p><p><strong>This article is part of an ongoing series exploring Agentic AI through progressively more capable multi-agent systems. Previous articles in the series include:</strong></p><ul><li><a href="https://medium.com/@icaro_vazquez/agentic-ai-a-quick-overview-an-example-3aa538926e53"><em>Agentic AI: A Quick Overview &amp; an Example</em></a> — introducing the basic concepts behind Agentic AI.</li><li><a href="https://medium.com/@icaro_vazquez/agentic-ai-a-more-generic-and-more-useful-example-9140b73027d4"><em>Building an Agentic AI Research Team</em> </a>— exploring how specialized agents can collaborate on research tasks.</li><li><a href="https://medium.com/@icaro_vazquez/what-observability-revealed-about-my-agentic-ai-team-15e442e6a6d7"><em>What Observability Revealed About My Agentic AI Team</em> </a>— using Langfuse to understand how agents behave, consume tokens, and interact within a multi-agent workflow.</li><li><a href="https://medium.com/@icaro_vazquez/building-an-agentic-ai-data-science-team-with-llms-a-multi-agent-machine-learning-workflow-2c6d38ed0f9b"><em>Building an Agentic AI Data Science Team with LLMs: A Multi-Agent Machine Learning Workflow</em></a> — extending the architecture into an autonomous machine learning team and establishing the foundation for the experiments in this article.</li></ul><p>This article continues that progression by asking a different question: <strong>can the architecture generalize?</strong></p><p>The previous article answered a fundamental question:</p><p><strong>Can we build an autonomous AI Data Science workflow?</strong></p><p>The answer was yes. But once the excitement of the first successful competition wore off, a much harder engineering question emerged.</p><p><strong>Can the same architecture solve completely different machine learning problems without being rewritten?</strong></p><p>I originally expected memory — allowing agents to learn from previous experiments — to be the next major addition. But before teaching the system to learn across experiments, I realized I needed to establish whether the architecture could handle fundamentally different experiments in the first place.</p><p>Answering that question became the real objective of the project.</p><p>Rather than optimizing for a single Kaggle competition, I treated the framework as an engineering experiment. The goal was to determine whether the system could adapt to new datasets, new prediction targets, and even different machine learning tasks by changing only a competition configuration file.</p><p>Over several iterations, the framework successfully completed three substantially different competitions:</p><ul><li>Titanic (binary classification)</li><li>House Prices (regression)</li><li>Spaceship Titanic (binary classification with a completely different schema)</li></ul><p>Each competition exposed weaknesses that forced changes to the architecture. Along the way, I removed competition-specific assumptions, generalized parts of the execution pipeline, and expanded the observability layer to compare LLM usage across experiments.</p><p>This article describes what worked, what didn’t, and how the framework evolved toward a more reusable Agentic AI system.</p><p><strong>The complete implementation is available on GitHub.</strong></p><p>The <a href="https://github.com/icarovazquez/agentic-ai-data-science-team/blob/main/notebooks/Generalized%20Kaggle%20Data%20Science%20Agentic%20AI%20Team.ipynb">repository</a> contains the evolving Agentic AI Data Science Team described in this article, including the specialized agents, deterministic executors, competition configuration system, experiment tracking, observability instrumentation, and the Kaggle workflows used for the experiments discussed below.</p><p>Readers interested in the implementation can follow the code alongside the architectural discussion in this article.</p><h3>Building a Framework instead of a Notebook</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sHQjYdi_QEwORKCbCHOZTg.png" /><figcaption><strong>Figure 1. One architecture, multiple machine learning problems. Competition-specific information enters through configuration, while the underlying Agentic AI workflow remains unchanged.</strong></figcaption></figure><p>Most Kaggle solutions are designed to solve exactly one problem.</p><p>A notebook is written for a specific dataset, a specific prediction target, and a specific evaluation metric. Column names are hard-coded, preprocessing is tightly coupled to the data, and model selection is tailored to the competition. When the next competition begins, most competitors simply create another notebook and start over.</p><p>There is nothing inherently wrong with this approach. Kaggle rewards leaderboard performance, not software architecture.</p><p>My objective was different. I wasn’t trying to build the best solution for a single competition. I wanted to know whether an AI team could solve many different machine learning problems while keeping the underlying architecture unchanged.</p><p>That distinction influenced every design decision that followed.</p><p>Instead of asking,</p><blockquote>“How do I solve the Titanic competition?”</blockquote><p>the framework asks,</p><blockquote>“Given this competition configuration, how should an AI Data Science team approach the problem?”</blockquote><p>That subtle shift changes everything. Rather than encoding competition-specific logic throughout the codebase, I concentrated the differences into a single configuration object, shown below.</p><pre>CompetitionConfig(<br>    competition_name=&quot;house-prices-advanced-regression-techniques&quot;,<br>    target_column=&quot;SalePrice&quot;,<br>    id_column=&quot;Id&quot;,<br>    prediction_column=&quot;SalePrice&quot;,<br>    problem_type=&quot;regression&quot;,<br>    evaluation_metric=&quot;root_mean_squared_log_error&quot;,<br>)</pre><p>Everything else remains the same.</p><p>Whether the framework is predicting passenger survival, estimating house prices, or determining whether passengers were transported to another dimension, the workflow follows the same sequence of reasoning and execution.</p><p>The agents never reason about “Titanic” or “House Prices.” They reason about a machine learning problem described through configuration.</p><p>That level of abstraction turned out to be one of the most important architectural decisions in the project. It allowed me to treat each Kaggle competition as another experiment rather than another software project.</p><h3>Reasoning is different from Execution</h3><p>As I described in the previous article, one of the foundational decisions in the architecture was separating reasoning from execution. The LLM agents decide what should happen; deterministic Python executors perform the actual machine learning operations.</p><p>That separation became even more important as I generalized the framework. It allowed the reasoning layer and execution layer to evolve independently: agents could adapt their recommendations to a new competition while the underlying executors remained reusable. Rather than revisit that architecture here, I want to focus on what happened when I began testing its limits.</p><h3>Specialized Agents Instead of One Generalist</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hiAZrlE5pPUagx2QxW6Qog.png" /><figcaption><strong>Figure 2. The Agentic AI Data Science Team separates specialized LLM reasoning agents from deterministic Python executors.</strong></figcaption></figure><p>One of the earliest design decisions was to avoid creating a single “super agent” responsible for solving an entire machine learning competition.</p><p>At first glance, a general-purpose agent seems attractive. Modern LLMs are capable of understanding datasets, proposing features, selecting algorithms, and explaining model performance. Asking one agent to perform all of those tasks would certainly simplify the architecture.</p><p>In practice, however, it quickly became difficult to manage. As prompts grew longer, they became harder to maintain. Responsibilities began to overlap, reasoning became less structured, and debugging became increasingly challenging. A change intended to improve one part of the workflow often produced unintended consequences somewhere else.</p><p>Instead, I adopted the same strategy used by many successful software engineering teams: specialization.</p><p>Each agent owns a single stage of the workflow:</p><ul><li>The Problem Framing Agent identifies the type of machine learning problem and defines success.</li><li>The Data Analysis Agent examines the dataset, identifies potential data quality issues, and proposes opportunities for feature engineering.</li><li>The Feature Engineering Agent recommends transformations, derived variables, encoding strategies, and preprocessing policies.</li><li>The Modeling Agent selects candidate algorithms appropriate for the competition.</li><li>Finally, the Evaluation Agent determines how those models should be validated and under what conditions they should be approved.</li></ul><p>None of these agents train a model. None perform cross-validation. None write submission files.</p><p>Their role is purely to reason.</p><p>That distinction keeps prompts focused and gives every agent a well-defined responsibility.</p><p>Perhaps more importantly, it also makes the system easier to improve. If the Feature Engineering Agent consistently produces weak recommendations, I can refine that prompt without affecting the Modeling Agent or the Evaluation Agent. Each component evolves independently.</p><p>Over time, this modularity proved far more valuable than trying to build a single all-knowing agent.</p><h3>Generalization Is an Architectural Problem</h3><p>When people talk about improving Agentic AI systems, the conversation usually focuses on prompts.</p><p>Prompt engineering has become almost synonymous with improving AI applications. If an agent produces poor results, the instinct is to rewrite the prompt, add more examples, or provide additional instructions.</p><p>I followed that same path early in this project.</p><p>Many iterations involved refining prompts, clarifying responsibilities, and improving the instructions given to each agent. Those changes certainly helped, but something interesting happened as the framework matured.</p><p>The largest improvements did not come from better prompts. They came from changing the architecture.</p><p>Each time the framework struggled with a new competition, the underlying cause was rarely that an agent lacked intelligence. More often, the architecture had embedded an assumption that was no longer true.</p><p>For example, moving from the Titanic competition to House Prices exposed assumptions about classification metrics, model selection, evaluation policies, and submission validation. None of those issues were solved by making the prompts longer or more sophisticated.</p><p>They were solved by removing competition-specific assumptions from the software itself.</p><p>The same pattern appeared again with Spaceship Titanic.</p><p>The Feature Engineering Agent generated useful recommendations, but the execution layer had to accommodate more complex preprocessing, high-cardinality categorical variables, and richer feature transformations. Once again, the solution was not to ask the agent to “reason harder.” It was to make the framework more flexible.</p><p>That experience changed how I think about Agentic AI systems.</p><p>Prompt engineering remains important. But prompts are not the primary mechanism for achieving generalization.</p><p>Generalization emerges when the architecture is modular, configurable, and free of unnecessary assumptions.</p><p>The LLMs provide intelligence. The software architecture determines how effectively that intelligence can be reused.</p><p>For me, that became the central lesson of the project.</p><h3>Kaggle Became the Laboratory, Not the Destination</h3><p>As the framework became more general, my objective for the project began to change.</p><p>Initially, Kaggle was the destination. I wanted to know whether an Agentic AI Data Science Team could autonomously work through a machine learning competition and produce a valid submission.</p><p>Once that worked, however, the competitions became useful for something much more interesting. They became experiments. Each new competition gave me a controlled environment for testing the architecture against a problem it had not been specifically designed to solve.</p><p>This is one of the reasons Kaggle turned out to be such a useful environment for developing the system. A competition provides a defined dataset, prediction target, evaluation metric, and independent scoring mechanism. That means I can change the architecture and observe what happens without changing the fundamental rules of the experiment.</p><p>More importantly, different competitions allow me to test different dimensions of generalization.</p><p>Titanic could test whether the basic architecture worked.</p><p>House Prices could test whether the same architecture could move from classification to regression.</p><p>Spaceship Titanic could test whether the framework could return to classification while dealing with a completely different schema and a richer feature engineering problem.</p><p>The question was therefore no longer simply:</p><blockquote>How well can the system solve this Kaggle competition?</blockquote><p>It became:</p><blockquote>What does this competition reveal about the architecture?</blockquote><p>That changed how I evaluated both successes and failures.</p><p>A failure was no longer necessarily a bad result. If a competition exposed a hidden assumption in the framework, it had identified something that needed to be generalized.</p><p>Likewise, a successful Kaggle submission was no longer sufficient evidence that the system was working well. A hard-coded workaround might improve the score while making the architecture less reusable.</p><p>The objective became progressively clearer:</p><p><strong>Improve the system in ways that make the next unknown problem easier to solve.</strong></p><p>That is a very different optimization target from maximizing the score of a single Kaggle notebook.</p><p>And it led to one of the most useful experiments in the project: running the same Agentic AI team against three different machine learning problems.</p><h3>Three Competitions, Three Architectural Lessons</h3><p>The three competitions were deliberately different enough to challenge different parts of the framework.</p><p>They also produced a useful progression.</p><h3>Titanic: Proving the Architecture</h3><p>Titanic was the starting point.</p><p>It is a relatively small binary classification problem with a familiar schema and well-understood feature relationships. That made it an ideal environment for validating the first version of the Agentic AI Data Science Team.</p><p>The important result was not the Kaggle score. It was that the complete workflow could execute.</p><p>The Problem Framing Agent identified the task as binary classification. The Data Analysis and Feature Engineering Agents reasoned about the dataset. The Modeling Agent recommended candidate models. Python executors transformed the data, trained and evaluated models, and ultimately generated a valid Kaggle submission.</p><p>The experiment demonstrated that the architecture was viable. But it did <strong>not</strong> demonstrate generalization.</p><p>In retrospect, this is an important distinction. A system that successfully solves the problem it was designed around may simply contain a sophisticated collection of assumptions about that particular problem.</p><p>The only way to discover those assumptions is to change the problem.</p><p>House Prices did exactly that.</p><h3>House Prices: Breaking the Classification Assumptions</h3><p>House Prices changed the nature of the machine learning task. Instead of predicting a binary outcome, the framework now had to estimate a continuous value: the sale price of a house.</p><p>That seemingly simple change exposed assumptions throughout the system:</p><ul><li>Classification metrics were no longer appropriate.</li><li>The candidate model set had to change.</li><li>Cross-validation had to evaluate regression performance.</li><li>Ensemble logic had to work with regressors instead of classifiers.</li><li>The submission pipeline had to preserve continuous predictions.</li></ul><p>One of the most revealing failures appeared in a surprisingly mundane place: <strong>the run leaderboard</strong>.</p><p>The original framework had been built around classification metrics such as accuracy, where a higher score represents a better model. That assumption had quietly found its way into the leaderboard ranking logic.</p><p>House Prices uses <strong>Root Mean Squared Logarithmic Error (RMSLE)</strong>. For RMSLE, <strong>lower</strong> scores represent better models.</p><p>As a result, the leaderboard ranked the worst regression models above the best ones. The problem wasn’t House Prices. The problem was that the architecture had implicitly defined <em>“best model”</em> as <em>“largest metric.”</em></p><p>Rather than adding a special case for House Prices, I redesigned the ranking logic so that the direction of comparison is determined from the configured evaluation metric. The leaderboard now understands that some metrics should be maximized while others should be minimized, allowing the same experiment tracking system to support both classification and regression competitions.</p><p>That small bug captured the larger lesson of the entire experiment.</p><blockquote><strong><em>Generalization failures often hide inside assumptions that seem perfectly reasonable when only one problem has been tested.</em></strong></blockquote><p>The solution was not to create a special “House Prices mode.” That would simply have replaced one hard-coded assumption with another.</p><p>Instead, the architecture had to understand the problem_type, evaluation metric, target column, and model requirements through configuration and propagate those decisions throughout the workflow.</p><p>Once those assumptions were removed, the same workflow could support both classification and regression.</p><p>That was the first meaningful evidence that the architecture was becoming more general.</p><h3>Spaceship Titanic: Could We Stop Changing the Workflow?</h3><p>The third experiment asked a subtly different question.</p><p>After the changes required by House Prices, could the framework solve another unfamiliar competition <strong>without another architectural redesign?</strong></p><p>Spaceship Titanic returned the system to binary classification, but almost everything else about the data was different.</p><p>The schema was different. The missing-value patterns were different. The categorical variables were different. And useful features had to be derived from entirely different relationships in the data.</p><p>This placed more pressure on the Data Analysis and Feature Engineering Agents.</p><p>And this time, something important happened.</p><p><strong>The workflow worked.</strong></p><p>I registered a new CompetitionConfig, pointed the framework at the new training and test datasets, and ran the same Agentic AI Data Science Team.</p><p>The agents adapted their reasoning to the new dataset while the executors carried out the resulting plan.</p><p>No Spaceship-Titanic-specific workflow was required.</p><p>That was a much more meaningful result than simply producing another successful Kaggle submission.</p><p>The framework had encountered a problem it had not originally been designed around and adapted through configuration and agent reasoning rather than through another set of hard-coded changes.</p><p>For me, that was the point at which the project began to feel less like a generalized Kaggle notebook and more like a reusable Agentic AI system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eq39eHigGoN-xatV656dzA.png" /><figcaption><strong>Figure 3. Three Kaggle Competitions served as controlled experiments for testing different dimensions of architectural generalization</strong></figcaption></figure><h3>What These Experiments Do — and Don’t — Prove</h3><p>There is an important limitation to this experiment.</p><p>Titanic, House Prices, and Spaceship Titanic are all relatively accessible tabular machine learning competitions. Together they test meaningful dimensions of generalization — classification versus regression, different schemas, different evaluation metrics, and different feature engineering requirements — but they do not represent the full range of machine learning problems.</p><p>The framework has not yet been tested against multiclass classification, severe class imbalance, time-series forecasting, natural language, image data, or competitions with more specialized evaluation metrics.</p><p>So I would not claim that three successful competitions demonstrate universal generalization.</p><p>What they demonstrate is narrower<strong> but still important: competition-specific assumptions can be progressively removed from an Agentic AI architecture, allowing the same workflow to operate across multiple tabular machine learning problems without being rewritten for each.</strong></p><p>The next meaningful test of the architecture should therefore be a competition that breaks one of the assumptions that still remains.</p><h3>Measuring the Cost of Generalization</h3><p>In the previous article, I used Langfuse to measure token consumption and cost <strong>within a single Agentic AI workflow</strong>. Here, I wanted to extend that idea across experiments. The question was no longer simply how much each agent consumed during one run, but whether different machine learning problems caused the same architecture to consume different amounts of LLM resources.</p><p>Completing three different Kaggle competitions demonstrated that the architecture could generalize.</p><p>But it raised another question:</p><blockquote><strong><em>Does every problem require the same amount of reasoning?</em></strong></blockquote><p>An Agentic AI workflow has two very different types of computational cost.</p><p>The deterministic layer consumes traditional computing resources while preprocessing data, training models, performing cross-validation, and tuning hyperparameters.</p><p>The reasoning layer consumes LLM resources.</p><p>Every time the Problem Framing Agent analyzes a competition, the Data Analysis Agent examines a dataset, or the Feature Engineering Agent develops a strategy, tokens are consumed.</p><p>Until recently, I could observe those individual calls through Langfuse, but I wasn’t treating LLM consumption as part of the experiment itself. That seemed like a missed opportunity.</p><p>If Kaggle competitions were going to serve as a laboratory for testing the architecture, then I wanted to measure not only whether the system solved a problem, but also how much LLM usage it required to get there.</p><p>So I added token tracking to the workflow.</p><p>Every LLM invocation now records three values:</p><ul><li>input tokens</li><li>output tokens</li><li>total tokens</li></ul><p>At the end of a run, those values are aggregated by agent.</p><p>A typical execution now produces a summary like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*egxwcYd9mzKYJIQZx1qomA.png" /><figcaption><strong>Figure 4. LLM token usage for a complete workflow run, broken down by agent.</strong></figcaption></figure><p>This provides two levels of visibility:</p><ul><li>At the <strong>agent level</strong>, I can see which parts of the workflow consume the most LLM resources.</li><li>At the <strong>run level</strong>, I can measure the total token consumption required to solve a competition.</li></ul><p>But simply printing those numbers wasn’t enough. I wanted them to become part of the experiment history.</p><h3>From a Kaggle Leaderboard to an AI Experiment Leaderboard</h3><p>The framework already maintained a run history containing information such as the competition, selected model, submission status, and eventually Kaggle scores.</p><p>I extended that history to include LLM token consumption as shown in Figure 5 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJunmPKiWgiB15SmKQdXlQ.png" /><figcaption><strong>Figure 5. The run leaderboard extends traditional machine learning experiment tracking with LLM token consumption. Each competition can now be compared not only by model performance, but also by the amount of LLM usage required to complete the workflow.</strong></figcaption></figure><p>Even with only three competitions, the first results are interesting. The latest Titanic run consumed approximately 12,900 LLM tokens, Spaceship Titanic approximately 15,200, and House Prices approximately 20,700. House Prices therefore required roughly 60% more LLM token consumption than Titanic in these runs. With only three competitions, it would be premature to conclude that regression problems or larger datasets inherently require more LLM resources. But the framework now provides the instrumentation needed to investigate that question systematically.</p><p>Each completed run now stores:</p><ul><li>competition</li><li>final model</li><li>ensemble used</li><li>CV score</li><li>Kaggle score</li><li>LLM total tokens</li><li>timestamp</li></ul><p>The run leaderboard can therefore compare not only <strong>machine learning performance</strong>, but also the <strong>reasoning resources used to achieve that performance</strong>.</p><p>And as more competitions are added, an even more interesting question emerges:</p><blockquote><strong><em>Are some machine learning problems inherently more expensive for an Agentic AI system to reason about than others?</em></strong></blockquote><p>Perhaps a relatively simple binary classification problem requires very little reasoning, while a dataset containing dozens of heterogeneous features requires substantially more analysis and feature engineering.</p><p>Or perhaps dataset complexity is less important than the quality of the prompts and context passed between agents.</p><p>I don’t know the answer yet. But now the system can measure it.</p><h3>What These Experiments Teach Us About Generalizable Agentic AI</h3><p>After running the same Agentic AI Data Science Team against three different machine learning problems, several broader lessons began to emerge.</p><p>Some were specific to this project, but others apply to Agentic AI systems far beyond data science.</p><p>The most important lesson was also the simplest:</p><blockquote><strong><em>Generalization is a property of the system, not the LLM.</em></strong></blockquote><p>Modern LLMs are already remarkably general-purpose reasoning engines. They can analyze a passenger manifest, reason about residential real estate, or interpret an unfamiliar dataset with surprisingly little additional instruction.</p><p>The harder problem is designing the software around them so that this reasoning can be reused.</p><p>That distinction became increasingly clear as the framework evolved.</p><h3>Remove Assumptions Instead of Adding Exceptions</h3><p>House Prices provided perhaps the clearest example.</p><p>When the framework encountered regression for the first time, the easiest solution would have been to add competition-specific logic:</p><pre>if competition == &quot;house-prices&quot;:<br>    ...</pre><p>That would have solved the immediate problem. It also would have made the framework less general. Instead, each failure became an opportunity to ask a different question:</p><blockquote>What assumption did the architecture make that is no longer valid?</blockquote><p>If model selection assumed classification, the solution was to make model selection depend on problem_type.</p><p>If evaluation assumed accuracy, the solution was to make the evaluation metric configurable.</p><p>If ranking assumed that higher scores were always better, the solution was to understand the direction of the configured metric.</p><p>This produced a useful engineering rule:</p><blockquote><strong><em>When a new problem breaks the system, remove the underlying assumption rather than add an exception for the new problem.</em></strong></blockquote><p>The difference is subtle, but it compounds. An exception helps solve the current problem. Removing an assumption helps solve problems you have not encountered yet.</p><h3>Configuration Defines the Problem; Architecture Defines the Process</h3><p>The CompetitionConfig object eventually became much more important than I originally expected.</p><p>It defines what changes from one experiment to another:</p><ul><li>dataset paths</li><li>target column</li><li>identifier column</li><li>problem type</li><li>evaluation metric</li><li>prediction column</li><li>submission requirements</li></ul><p>The workflow defines what remains constant.</p><p>This creates a clean boundary between <strong>problem definition</strong> and <strong>problem-solving process</strong>.</p><p>Adding Spaceship Titanic therefore did not require creating another workflow. It required describing another problem.</p><p>That distinction is fundamental to generalization. A reusable Agentic AI system should not need to know every problem it might encounter in advance. It needs a sufficiently expressive way to describe a new problem when it arrives.</p><h3>Structured Interfaces Matter More as the System Grows</h3><p>Another lesson emerged from the interactions between agents and executors.</p><p>Natural language works extremely well for communicating with humans. It is considerably less reliable as an interface between software components.</p><p>As the workflow became more sophisticated, agent outputs increasingly needed to conform to explicit structures. The Feature Engineering Agent, for example, could not simply recommend in prose that two variables should be combined. The executor needed to know the operation, source columns, parameters, and execution stage.</p><p>The output therefore became a contract between the reasoning layer and the execution layer.</p><p>This made JSON parsing and schema validation unexpectedly important parts of the architecture.</p><p>It also exposed one of the practical challenges of building Agentic AI systems: LLMs can generate excellent reasoning while still producing malformed structured output.</p><p>Some of the most frustrating failures in this project had nothing to do with machine learning. They were missing JSON keys, truncated responses, invalid syntax, or transformations described elegantly in prose that an executor could not actually perform.</p><p>The lesson was clear:</p><blockquote><strong><em>The more autonomous the system becomes, the more explicit the contracts between its components need to be.</em></strong></blockquote><p>Agents can remain probabilistic internally. Their interfaces should not.</p><h3>Generalization Should Be Tested, Not Assumed</h3><p>Perhaps the easiest mistake to make after building a successful agent is to call it general. I made that mistake after Titanic.</p><p>The system had completed an entire machine learning workflow, so it felt reusable. House Prices demonstrated otherwise.</p><p>A system has not generalized simply because its prompts avoid mentioning a specific dataset.</p><p>Generalization has to be demonstrated by exposing the architecture to problems that violate its previous assumptions.</p><p>That is why the progression across the three competitions mattered:</p><ul><li>Titanic demonstrated viability.</li><li>House Prices exposed hidden assumptions.</li><li>Spaceship Titanic tested whether the resulting abstractions actually worked.</li></ul><h3>Efficiency Is Part of Generalization</h3><p>Finally, the experiments introduced a dimension I had initially overlooked.</p><p>A system may be capable of solving many problems while becoming increasingly expensive to operate.</p><p>If every new problem requires dramatically larger prompts, longer reasoning chains, or substantially more LLM calls, the architecture may technically generalize while becoming economically impractical.</p><p>That is why token consumption is now part of the experiment history.</p><p>The objective is no longer simply:</p><blockquote>Can the system solve the problem?</blockquote><p>Eventually, the better question may be:</p><blockquote><strong><em>How effectively can the system solve the problem relative to the reasoning resources it consumes?</em></strong></blockquote><p>That opens a new set of experiments around model selection, prompt efficiency, latency, and cost. And it suggests that generalization and efficiency should ultimately be evaluated together.</p><h3>A Different Way to Think About Agentic AI</h3><p>The biggest change produced by these experiments may have been conceptual.</p><p>I started the project thinking primarily about agents:</p><ul><li>Which agents should exist?</li><li>What should their prompts say?</li><li>Which model should power them?</li></ul><p>I now think much more about the system around those agents:</p><ul><li>How are problems represented?</li><li>What assumptions are embedded in the architecture?</li><li>How do components communicate?</li><li>What happens when structured output fails?</li><li>Can behavior be measured?</li><li>Can experiments be compared?</li><li>Can the system encounter a new problem without requiring another branch in the code?</li></ul><p>Those questions sound much more like traditional software engineering than prompt engineering.</p><p>I don’t think that’s accidental.</p><p>As Agentic AI systems become more capable, the challenge increasingly shifts from getting an LLM to produce an intelligent answer to building reliable software systems that can repeatedly harness that intelligence.</p><p>The LLM may be the most sophisticated component in the architecture. But it is still only <em>one</em> component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2OxupkCJH2Cv5Dc9uDWttQ.png" /><figcaption><strong><em>Figure 6. </em>The evolution from prompt-centric AI to system-centric Agentic AI. Generalization emerges not from a single prompt or model, but from the architecture surrounding the LLM.</strong></figcaption></figure><h3>Conclusion: From Building Agents to Engineering Agentic Systems</h3><p>I no longer think of the competitions primarily as problems to solve. I think of them as experiments designed to break the architecture in useful ways.</p><p>Each failure exposes an assumption. Each assumption that can be removed makes the framework more general. And each new competition provides another opportunity to test whether those abstractions actually hold.</p><p>Increasingly, I believe the more difficult engineering challenge lies in everything surrounding the LLM:</p><ul><li>How do we represent a new problem?</li><li>How do agents communicate decisions to deterministic software?</li><li>How do we prevent competition-specific assumptions from leaking into reusable components?</li><li>How do we observe what the system is doing?</li><li>How do we compare one experiment with another?</li><li>And eventually, how do we determine whether the additional reasoning performed by an AI system is actually worth its computational cost?</li></ul><p>These are not primarily prompt-engineering questions. They are systems-engineering questions.</p><p>And that may be the broader lesson from this experiment.</p><blockquote><strong><em>The challenge is no longer simply building intelligent agents. It is engineering systems that can reliably reuse that intelligence across problems they have never encountered before.</em></strong></blockquote><p>The Agentic AI Data Science Team is still a work in progress. There are many more competitions to test, more assumptions to uncover, and more parts of the architecture to improve.</p><p>But now the framework itself provides the environment for running those experiments.</p><p>And it also raises the next question.</p><p>So far, the agents have largely relied on the same family of language models. But if the architecture is truly modular, there is no reason every agent needs the same brain.</p><p>Could a smaller and cheaper model perform Problem Framing just as well?</p><p>Does Feature Engineering benefit enough from a more capable model to justify the additional cost?</p><p>Can open-source models compete with proprietary frontier models inside the same workflow?</p><p>Could some — or eventually all — of the Agentic AI Data Science Team run locally on a laptop?</p><p>Those will be the next experiments.</p><p><strong>Code:</strong> The complete project and the latest version of the Agentic AI Data Science Team are available in the <a href="https://github.com/icarovazquez/agentic-ai">Agentic AI GitHub repository</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4107bc75f5cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Agentic AI Data Science Team with LLMs: A Multi-Agent Machine Learning Workflow]]></title>
            <link>https://medium.com/@icaro_vazquez/building-an-agentic-ai-data-science-team-with-llms-a-multi-agent-machine-learning-workflow-2c6d38ed0f9b?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c6d38ed0f9b</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 03:46:46 GMT</pubDate>
            <atom:updated>2026-07-01T06:03:17.078Z</atom:updated>
            <content:encoded><![CDATA[<p>How I designed a multi-agent system that frames machine learning problems, engineers features, trains and evaluates models, performs hyperparameter tuning, tracks experiments &amp; automatically generates Kaggle submissions.</p><h3>Introduction</h3><p>This article is the latest step in a journey that began several months ago.</p><p>In my <a href="https://medium.com/@icaro_vazquez/improving-my-agentic-ai-team-how-better-contracts-structured-outputs-and-evaluations-improved-6dc0651cb287">previous articles</a>, I explored the idea of building Agentic AI Teams rather than individual AI agents; my Agentic AI Research Team is depicted in Figure 1 below. Instead of asking one Large Language Model (LLM) to solve an entire problem, I created specialized agents with clearly defined responsibilities: a project manager, researchers, technical writers, editors, and evaluators collaborating to produce higher-quality work than a single prompt could achieve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HShhr39_r6g8Xb0F.png" /><figcaption><strong>Figure 1. The original Agentic AI Research Team introduced in my previous articles. This project extends the same architectural principles into the domain of machine learning.</strong></figcaption></figure><p>Those projects taught me valuable lessons about agent collaboration, structured outputs, agent contracts, and observability. Each iteration made the team more reliable and easier to understand.</p><p>But there was a fundamental limitation. The output of the research team is inherently subjective.</p><p>Even with evaluation agents and structured rubrics, it’s difficult to determine whether one report is better than another. Improvements often rely on human judgement, making it challenging to know whether architectural changes actually improved the system or produced different results.</p><p>I wanted to move to a domain where every architectural decision would be measured objectively.</p><p>Data Science turned out to be the perfect environment.</p><p>Unlike writing research reports, machine learning provides clear, quantitative feedback. Every feature engineering decision, every model choice, every parameter configuration, and every experiment can be evaluated using reproducible metrics like cross-validation accuracy, F1 score, ROC AUC, and, ultimately, Kaggle Leaderboard performance.</p><p>That changed the nature of the project entirely.</p><p>After building a team that produces research reports, I built an Agentic AI Data Science Team that reasons about machine learning problems, executes deterministic workflows, compares competing models, performs hyperparameter tuning, analyzes feature importance, runs ablation studies, evaluates ensembles, tracks experiments, and, ultimately, generates production-ready Kaggle submissions.</p><p>More importantly, every improvement to the system can now be measured.</p><p>When I change the architecture, I can observe whether cross-validation improves. When I introduce hyperparameter tuning, I can measure the effect on leaderboard performance. When I test feature ablations or ensemble methods, the results are reproducible rather than subjective.</p><p>In this article, I’ll walk through the architecture of the system, explain the design decisions behind each specialized agent and deterministic executor, and share engineering lessons I learned while building an autonomous data science team capable of solving end-to-end machine learning problems.</p><h3>System Architecture</h3><p>At a high level, the Agentic AI Data Science team is organized around a simple design principle:</p><p><strong>Reasoning and execution should be separated.</strong></p><p>Large Language Models excel at reasoning, planning, comparing alternatives, and making recommendations. One of the biggest mistakes I see in Agentic AI projects is asking a Language Model to do everything. The LLM is expected to analyze data, decide what to do next, write code, execute it, evaluate the results, decide if the results are good enough, and continue iterating until the task is complete.</p><p>While this can work for simple demonstrations, it quickly becomes difficult to reproduce, debug, and extend as the system grows. Instead, I chose a different architecture.</p><p>Large Language Models excel at reasoning. They are very good at understanding context, comparing alternatives, evaluating trade-offs, and making recommendations. They are not responsible for deterministic computation.</p><p>Training machine learning models, calculating evaluation metrics, performing cross-validation, generating Kaggle submission files, and storing experiment history are all deterministic tasks that should always produce the same results given the same inputs.</p><p>For those deterministic tasks, I rely on traditional Python executors.</p><p>This leads to a simple division of responsibilities:</p><ul><li><strong>Reasoning Agents</strong> use LLMs to analyze information and make decisions.</li><li><strong>Execution Executors</strong> perform deterministic machine learning operations.</li><li><strong>Shared Records</strong> allow every component to communicate through structured contracts rather than free-form text.</li></ul><p>The last design decision turned out to be one of the most important ones of the project.</p><p>Rather than passing natural language between components, every stage produces a structured record that becomes the input for the next component. These records constitute explicit contracts between components, making the workflow reproducible, modular, and easy to extend.</p><p>The complete architecture is shown in Figure 2 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MQycLRpe0SkyAY9isJqByQ.png" /><figcaption><strong>Figure 2. The Agentic AI Data Science Team alternates between reasoning agents and deterministic executors. Structured records act as contracts between every stage of the workflow.</strong></figcaption></figure><p>Notice that the workflow alternates between reasoning and execution. The reasoning agents determine <strong>what</strong> <strong>should happen next</strong>. The executors determine <strong>what</strong> <strong>actually happens</strong>.</p><p>This separation allows the LLM to focus exclusively on decision-making while leaving numerical computation, model training, experiment tracking, and reproducibility to deterministic Python code.</p><p>As the workflow evolved, another useful abstraction emerged: a centralized Model Factory. Rather than embedding model construction logic inside multiple executors, every component requests models through a shared factory. This ensures that hyperparameters, preprocessing pipelines, and model definitions remain consistent throughout the workflow while making it straightforward to introduce new algorithms without modifying existing executors.</p><p>As the project evolved, this architecture made it remarkably easy to introduce new capabilities — including hyperparameter tuning, feature importance analysis, feature ablation studies, ensemble learning, experiment persistence, dashboards &amp; a centralized model factory — without changing the overall workflow.</p><h3>Observability as a First-Class Design Principle</h3><p>One lesson carried directly from previous Agentic AI projects was that observability cannot be an afterthought. Figure 3 below shows part of the Agentic AI Data Science Team trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u7DbdiXgl1GK-ngPP6a7dg.png" /><figcaption><strong>Figure 3. Langfuse trace showing a partial execution of the Agentic AI Data Science Team, including reasoning agents, executors, token usage, and execution timeline.</strong></figcaption></figure><p>As the number of specialized agents increased, understanding what each agent was doing was just as important as the final machine learning results.</p><p>Every reasoning agent and executor in the system is instrumented using Langfuse. Each execution produces a complete trace, showing what components were called, the prompts and structured outputs exchanged between them, token usage, execution order, and overall workflow cost.</p><p>This provides two important capabilities:</p><ul><li>First, it makes debugging dramatically easier. If an agent makes an incorrect recommendation or produces an unexpected structured record, I can inspect the exact reasoning path that produced this result.</li><li>Second, it allows me to measure the operational cost of the entire workflow. Every run records input tokens, output tokens, estimated API cost, and per-agent usage statistics, making it possible to evaluate not only model performance but also the computational cost of reaching that result.</li></ul><p>In practice, I was optimizing two different systems simultaneously:</p><ul><li>The machine learning pipeline, measured by validation metrics &amp; Kaggle scores.</li><li>The Agentic AI work itself, measured by traces, latency, token consumption, and execution cost.</li></ul><p>Observability transformed the project from a collection of AI prompts into an engineering system that could be monitored, analyzed, and continuously improved.</p><h3>From Research Teams to Data Science Teams</h3><p>Although the application changed completely, most of the original architecture remained intact.</p><p>The Research Team I introduced in my previous articles and the Data Science Team share the same underlying philosophy:</p><ul><li>Specialized reasoning agents</li><li>Structured outputs</li><li>Explicit contracts between components</li><li>Deterministic execution</li><li>End-to-end observability</li></ul><p>What changed wasn’t the architecture. It was the domain.</p><p>The original research team was designed to answer open questions by coordinating several specialized agents. A project manager decomposed the problem into smaller research tasks, researchers gathered information using external tools, a technical writer assembled the findings into a report, an editor improved clarity and consistency, and evaluation agents assessed the quality of the final result.</p><p>That architecture worked remarkably well for knowledge work.</p><p>However, it also exposed an important limitation.</p><p>Research quality is inherently subjective.</p><p>Even with structured rubrics and evaluation agents, there is rarely a single “correct” answer. One report may be more comprehensive, another more concise, and another more persuasive. Architectural improvements were often difficult to validate because the evaluation itself relied on human judgment.</p><p>Machine learning presents a fundamentally different optimization problem.</p><p>Instead of asking whether one report is better than another, we can ask objective questions:</p><ul><li>Did cross-validation accuracy improve?</li><li>Did the model generalize better?</li><li>Did hyperparameter tuning outperform the baseline?</li><li>Did removing a feature improve predictive performance?</li><li>Did the ensemble outperform the best individual model?</li><li>Did the Kaggle leaderboard score increase?</li></ul><p>Every architectural decision now has measurable consequences.</p><p>This shifted the role of the agents.</p><p>Rather than coordinating research activities, each agent became responsible for reasoning about one stage of the machine learning lifecycle:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*li4YVwf_iNb-N2ZyjuBtZQ.png" /></figure><p>The transformation went beyond simply renaming the agents. The outputs became more structured.</p><p>Instead of producing paragraphs of text for another agent to interpret, every agent now produces a well-defined record describing its recommendations. These records act as contracts between reasoning agents and deterministic executors, making the entire workflow reproducible and significantly easier to debug.</p><p>This evolution also changed how I thought about the role of Large Language Models. In the research team, the LLM generated the primary artifact: the final report. In the Data Science Team, the LLM no longer produces the final deliverable.</p><p>Instead, it acts as an engineering advisor. It analyzes the problem, recommends feature-engineering strategies, proposes candidate models, evaluates competing approaches, and decides which experiments to perform next. Traditional Python code then executes those decisions deterministically.</p><p>That distinction turned out to be one of the most important architectural lessons of the project.</p><p>The language model reasons.</p><p>The software system executes.</p><p>By separating those responsibilities, I gained something difficult to achieve in the earlier research projects: the ability to measure, reproduce, and continuously improve every part of the workflow using objective evidence rather than subjective judgment.</p><h3>Why Titanic?</h3><p>You might wonder why I chose the Titanic competition for demonstrating this architecture.</p><p>The answer is pragmatic rather than competitive. My goal wasn’t to achieve a top Kaggle ranking or push the limits of predictive performance. My goal was to validate the architecture itself.</p><p>Titanic provides several advantages for that purpose:</p><ul><li>It is a well-understood binary classification problem.</li><li>The dataset is small enough to support rapid experimentation.</li><li>Training cycles complete in seconds rather than hours.</li><li>Every architectural change can be evaluated quickly using reproducible metrics.</li><li>The workflow exercises the complete machine learning lifecycle, from feature engineering through model selection, hyperparameter tuning, experiment tracking, and submission generation.</li></ul><p>In other words, Titanic served as a controlled engineering environment rather than the final destination.</p><p>The same architecture can be applied to substantially more complex machine learning problems, and extending the system to additional Kaggle competitions is one of the next steps in its evolution.</p><h3>The End-to-End Workflow</h3><p>With the architecture in place, let’s follow a machine learning problem as it moves through the Agentic AI Data Science Team.</p><p>The workflow begins long before a machine learning model is trained. The first responsibility of the system is to understand the problem itself.</p><p>The <strong>Problem Framing Agent</strong> receives the competition description and determines fundamental characteristics of the task:</p><ul><li>What is the prediction target?</li><li>Is this a classification or regression problem?</li><li>Which evaluation metric should be optimized?</li><li>What constitutes a successful solution?</li></ul><p>Rather than immediately writing Python code, the system first develops an understanding of the problem it is trying to solve. Figure 4 below shows the Problem Framing Record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dw87BHN-cn5V4zXNsIKINw.png" /><figcaption><strong>Figure 4. The Problem Framing Agent produces a structured record rather than free-form text. This record becomes the contract for downstream components.</strong></figcaption></figure><p>That structured understanding becomes the input to the <strong>Data Analysis Agent</strong>.</p><p>Using dataset metadata and exploratory statistics, this agent identifies potential issues and opportunities, including missing values, categorical variables, feature distributions, possible data leakage, and opportunities for feature engineering.</p><p>The <strong>Feature Engineering Agent</strong> then reasons about how the dataset can be improved.</p><p>Instead of modifying the data directly, it produces a structured specification describing the transformations that should be performed. These transformations and other fields are shown in Figure 5 below. The corresponding executor applies those transformations deterministically, ensuring every run remains reproducible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8xTm2h-_Cr1YBXcY1TzNLQ.png" /><figcaption><strong>Figure 5. The Feature Engineering Agent proposes transformations that are executed deterministically by Python.</strong></figcaption></figure><p>Once the engineered dataset has been created, the workflow transitions into model development.</p><p>The <strong>Modeling Agent</strong> recommends a collection of candidate algorithms appropriate for the problem. Rather than committing to a single model immediately, the system trains several competing approaches, including Logistic Regression, Random Forest, Gradient Boosting, and Extra Trees.</p><p>Each candidate model is evaluated using both a validation split and stratified cross-validation. Figure 6 below shows a partial cross-validation leaderboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tro3Jg7YxNDs4L0Y5Hmv6w.png" /><figcaption><strong>Figure 6. Candidate models are compared using stratified cross-validation before the Model Evaluation Agent selects the model that advances to the next stage.</strong></figcaption></figure><p>At this point, another reasoning agent enters the workflow.</p><p>The <strong>Model Evaluation Agent</strong> reviews the quantitative results and determines which model should move forward. Rather than selecting the highest single validation score, the agent considers consistency across folds, model stability, and the likelihood of generalization. A partial view of its output is shown in Figure 7 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EeNuNjaSdm9CZJQONs1MMA.png" /><figcaption><strong>Figure 7. The Model Evaluation Record shows which model will move forward</strong></figcaption></figure><p>Once a model has been selected, a series of deterministic optimization stages are executed.</p><p>The workflow performs:</p><ul><li>Hyperparameter tuning using grid search</li><li>Feature importance analysis</li><li>Feature ablation studies</li><li>Ensemble evaluation</li></ul><p>Each experiment is recorded automatically, allowing the system to compare architectural decisions using objective performance metrics rather than intuition.</p><p>Finally, the best-performing model is retrained using the complete labeled dataset before generating predictions for the Kaggle test set.</p><p>The workflow concludes by creating a properly formatted submission file, recording the experiment in a persistent history, updating the results dashboard, and capturing complete observability traces, token usage, and execution cost for the entire run.</p><p>By the time the pipeline finishes, the system has not only generated predictions — it has also documented every important engineering decision that led to those predictions. That capability ultimately became one of the project’s biggest advantages.</p><h3>Structured Contracts Between Components</h3><p>One of the most important architectural lessons from my previous Agentic AI projects was the value of <strong>structured contracts</strong> between agents.</p><p>In the earliest versions of the research team, agents primarily communicated through natural language. While that approach worked well for simple workflows, observability quickly revealed how fragile those interactions became as the system grew. Small variations in an agent’s output could make downstream components more difficult to interpret, debug, and evaluate.</p><p>As I described in my previous article, introducing structured outputs and explicit contracts between agents made the workflow significantly more reliable. Rather than exchanging free-form text, each agent produced a well-defined record describing its recommendations in a consistent format.</p><p>The Agentic AI Data Science Team builds directly on that architectural evolution.</p><p>Every reasoning agent produces a structured record as its primary output, allowing downstream components to consume structured data instead of interpreting free-form text. This record acts as an explicit contract between agents and deterministic executors. Instead of asking downstream components to interpret paragraphs of text, each stage of the workflow receives well-defined inputs and produces equally well-defined outputs.</p><p>For example, instead of asking the Feature Engineering Executor to interpret a paragraph describing possible transformations, the Feature Engineering Agent returns a structured object specifying exactly which transformations should be applied.</p><p>Likewise, the Model Evaluation Agent doesn’t simply explain why one model appears better than another. It produces an evaluation record that includes the recommended model, the rationale for that recommendation, and whether the model is ready to proceed to hyperparameter tuning.</p><p>A simplified version of the workflow looks like this:</p><pre>Problem Framing Agent<br>        │<br>        ▼<br>problem_framing_record<br>        │<br>        ▼<br>Data Analysis Agent<br>        │<br>        ▼<br>data_analysis_record<br>        │<br>        ▼<br>Feature Engineering Agent<br>        │<br>        ▼<br>feature_engineering_record<br>        │<br>        ▼<br>Feature Engineering Executor<br>        │<br>        ▼<br>engineered_dataset<br>        │<br>        ▼<br>Model Training Executor<br>        │<br>        ▼<br>model_training_record<br>        │<br>        ▼<br>Cross Validation Executor<br>        │<br>        ▼<br>cross_validation_record<br>        │<br>        ▼<br>Model Evaluation Agent<br>        │<br>        ▼<br>evaluation_record</pre><p>Every component knows exactly what information it should receive and exactly what information it should produce. For example, here’s what the problem framing record has:</p><pre>{&#39;competition_name&#39;: &#39;titanic&#39;, <br>  &#39;problem_type&#39;: &#39;binary_classification&#39;, <br>  &#39;prediction_target&#39;: &#39;Survived&#39;, <br>  &#39;evaluation_metric&#39;: &#39;accuracy&#39;, <br>  &#39;submission_columns&#39;: [&#39;PassengerId&#39;, &#39;Survived&#39;], <br>  &#39;success_criteria&#39;: &#39;Maximize accuracy of survival predictions on test set&#39;, <br>}</pre><p>This seemingly simple decision had several important consequences.</p><p>First, the workflow became much easier to debug. If an executor produced an unexpected result, I could inspect the corresponding record rather than searching through prompts or conversation history.</p><p>Second, the architecture became highly modular. Because each component communicates through a well-defined contract, new agents and executors can be added without changing the rest of the system.</p><p>Third, the workflow became significantly more reproducible. The reasoning performed by each agent is preserved as structured data, enabling inspection, replay, and comparison of decisions across experiments.</p><p>Finally, structured contracts dramatically simplified observability. Every record became a natural checkpoint within the workflow, making it straightforward to trace the evolution of a machine learning project from problem definition to final Kaggle submission.</p><p>Looking back, this architectural decision had a much bigger impact than any individual prompt improvement. Prompts can always be refined, but clear interfaces between components are what make complex agentic systems maintainable as they continue to grow.</p><h3>Experimentation and Continuous Improvement</h3><p>Building the first version of the Agentic AI Data Science Team was only the beginning.</p><p>The real objective was to create a system capable of improving itself through systematic experimentation rather than ad hoc trial and error.</p><p>Every execution of the workflow generates a complete experiment record containing:</p><ul><li>The model selected by the evaluation agent</li><li>Validation and cross-validation metrics</li><li>Hyperparameter tuning results</li><li>Feature importance analysis (example shown in Figure 8 below)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*qfNyxfyuYzLOkUnuLBRaHg.png" /><figcaption><strong><em>Figure 8.Feature importance analysis reveals which engineered features contribute most to predictive performance.</em></strong></figcaption></figure><ul><li>Feature ablation results (example shown in Figure 9 below)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4yL75uQmdn8WS-anNpMM2w.png" /><figcaption><strong>Figure 9. Ablation experiments quantify the contribution of engineered feature groups to overall model performance.</strong></figcaption></figure><ul><li>Ensemble evaluation</li><li>Kaggle submission information</li><li>Public leaderboard score</li><li>Timestamp and experiment metadata</li></ul><p>Rather than treating each Kaggle submission as an isolated attempt, the system maintains a persistent experiment history that records every architectural change together with its validation metrics, leaderboard performance, and experimental context. This allows each iteration of the workflow to be evaluated objectively and compared against previous approaches. For the Titanic competition, the progression looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZAtcjt9D0QWWcXH10zuEQQ.png" /></figure><p>Each experiment answered a specific engineering question:</p><ul><li>Would Gradient Boosting generalize better than Logistic Regression?</li><li>Would hyperparameter tuning improve cross-validation performance?</li><li>Would removing engineered features improve the model?</li><li>Would an ensemble outperform the strongest individual model?</li></ul><p>Some experiments improved performance.</p><p>Others demonstrated that seemingly reasonable ideas — such as replacing the tuned Gradient Boosting model with a soft voting ensemble — actually reduced predictive accuracy.</p><p>Those “negative” results proved just as valuable as the successful ones.</p><p>Instead of relying on intuition, every architectural decision became an experiment with measurable outcomes.</p><p>One interesting aspect of this architecture is that it naturally creates what many practitioners are now referring to as <strong>loop engineering</strong>. Rather than executing a workflow once and stopping, the system continuously feeds the results of each experiment back into the next iteration of the engineering process. Validation metrics, cross-validation results, feature ablation studies, hyperparameter searches, leaderboard scores, and experiment history all become inputs that guide future decisions.</p><p>In other words, the workflow doesn’t simply automate machine learning tasks — it creates a measurable feedback loop that allows the architecture itself to evolve over time.</p><p>Although the current implementation still relies on a human to review results and launch the next experiment, the underlying loop is already present. Extending the system so that agents automatically propose and execute the next experiment based on previous outcomes is a natural next step.</p><p>That philosophy fundamentally changed how I approached building agentic systems.</p><p>Rather than asking whether a prompt “felt better,” I could ask whether a new capability measurably improved model performance, reduced execution cost, or increased reproducibility.</p><p>One unexpected outcome of this project was realizing that software architecture itself became measurable. Every change to an agent, executor, or workflow could be evaluated objectively using validation metrics, experiment history, execution traces, and operational cost rather than intuition alone.</p><p>The project became an engineering process driven by evidence instead of iteration by instinct.</p><h3>Measuring the AI Team</h3><p>One of the most interesting lessons from this project was realizing that an Agentic AI system introduces a second optimization problem.</p><p>In traditional machine learning, engineers focus primarily on improving predictive performance. Models are evaluated using metrics such as validation accuracy, cross-validation scores, F1, ROC AUC, and ultimately leaderboard performance.</p><p>An Agentic AI workflow requires another layer of measurement. It’s no longer sufficient to ask whether the model improved. We should also ask whether <strong>the AI team itself</strong> became more efficient, more reliable, and easier to operate.</p><p>That means measuring the engineering process in addition to the machine learning results.</p><p>The Agentic AI Data Science Team captures operational metrics throughout every workflow execution, including:</p><ul><li>Input, output, and total token consumption</li><li>Estimated API cost</li><li>Per-agent token usage</li><li>Number of agent calls</li><li>Complete execution traces</li><li>Persistent experiment history</li></ul><p>Together, these metrics make it possible to optimize not only the quality of the predictions but also the efficiency of the AI workflow itself.</p><p>The dashboard, shown in Figure 10, brings these different dimensions together into a single view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*belJQqJPKnBzNntXsiwLLg.png" /><figcaption><strong>Figure 10. The results dashboard combines machine learning performance, experiment history, feature analysis, ensemble evaluation, and operational metrics into a single view.</strong></figcaption></figure><p>On one side, it summarizes machine learning performance, including validation accuracy, hyperparameter tuning results, feature importance analysis, feature ablation experiments, ensemble evaluation, experiment history, and Kaggle leaderboard performance. On the other hand, it reports operational metrics such as total token consumption, per-agent usage statistics, and estimated execution cost (Figure 11).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ygiU0mCTpTqloc1RIWAYag.png" /><figcaption><strong><em>Figure 11. Operational metrics collected during each workflow execution allow the engineering team to optimize not only prediction quality but also the efficiency of the Agentic AI system.</em></strong></figcaption></figure><p>While the dashboard provides a high-level summary of each workflow execution, observability allows the engineering process itself to be inspected in detail. Figure 12 shows a partial Langfuse trace of the Agentic AI Data Science Team, including execution order, agent interactions, token usage, and the reasoning produced by the Feature Engineering Agent during one stage of the workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qqw2AWtqtb2HhP11a2v7tg.png" /><figcaption><strong>Figure 12. Partial Langfuse trace of the Agentic AI Data Science Team showing execution order, agent interactions, token usage, and the detailed reasoning produced by the Feature Engineering Agent during one stage of the workflow.</strong></figcaption></figure><p>One interesting consequence of this architecture is that three different layers of the system become measurable.</p><p><strong>Machine Learning Performance</strong></p><ul><li>Validation Accuracy</li><li>Cross-Validation</li><li>F1 Score</li><li>ROC AUC</li><li>Kaggle Leaderboard Score</li></ul><p><strong>AI Workflow Performance</strong></p><ul><li>Token Usage</li><li>API Cost</li><li>Execution Timeline</li><li>Agent Interactions</li><li>Langfuse Traces</li></ul><p><strong>Architectural Evolution</strong></p><ul><li>Experiment History</li><li>Structured Contracts</li><li>Feature Engineering Experiments</li><li>Hyperparameter Tuning</li><li>Loop Engineering</li></ul><p>This turned out to be one of the most valuable outcomes of the project.</p><p>Rather than relying on intuition to determine whether the system was improving, nearly every aspect of the workflow became measurable. Model quality, operational efficiency, and even architectural decisions could be evaluated using objective evidence.</p><p>Looking back, I think this is where Agentic AI begins to resemble traditional software engineering. We’re no longer building prompts. We’re engineering systems that can reason, execute, measure their own performance, and continuously improve through evidence.</p><h3>Engineering Lessons Learned</h3><p>Building the Agentic AI Data Science Team changed several assumptions I had about designing AI systems.</p><p>Some of the most valuable lessons had very little to do with prompting language models. Instead, they came from applying familiar software engineering principles to Agentic AI.</p><h3>1. Specialization scales better than generalization</h3><p>Early Agentic AI examples often rely on a single agent responsible for solving an entire problem.</p><p>That approach works for simple demonstrations but becomes increasingly difficult to maintain as workflows become more complex.</p><p>Dividing responsibilities among specialized agents made the system easier to reason about, easier to test, and significantly easier to extend. Each agent could focus on one decision while deterministic executors handled the underlying computation.</p><h3>2. Reasoning and execution should remain separate</h3><p>This became the central architectural principle of the project.</p><p>Large Language Models are excellent at analyzing information, comparing alternatives, and making recommendations.</p><p>Traditional software is much better at deterministic computation.</p><p>Keeping those responsibilities separate produced a workflow that was both intelligent and reproducible.</p><h3>3. Structured contracts are more valuable than longer prompts</h3><p>One of the biggest improvements over my earlier projects came from treating every interaction between components as an explicit software contract.</p><p>Instead of exchanging paragraphs of text, agents exchanged structured records.</p><p>That single decision simplified debugging, enabled modular development, and made the workflow much easier to evolve over time.</p><h3>4. Observability is a requirement, not an enhancement</h3><p>As the number of agents increased, observability became essential.</p><p>Without execution traces, token usage, structured records, and workflow metrics, understanding why the system behaved the way it did would have been extremely difficult.</p><p>Observability transformed the project from an AI demo into an engineering system.</p><h3>5. Objective feedback accelerates architectural improvement</h3><p>Perhaps the most important lesson was choosing a domain where improvements could be measured objectively.</p><p>Unlike research reports, machine learning provides immediate quantitative feedback.</p><p>Every architectural change could be evaluated using cross-validation metrics, Kaggle leaderboard scores, experiment history, and operational measurements.</p><p>That feedback loop allowed the architecture itself to improve through evidence rather than intuition.</p><p>Looking back, I don’t think the most important artifact produced by this project is the Kaggle submission.</p><p>It’s the architecture.</p><p>The machine learning competition simply provided an environment where that architecture could be tested, measured, and refined.</p><h3>Where This Architecture Goes Next</h3><p>Although the Agentic AI Data Science Team can already solve end-to-end machine learning problems, I see this project as the beginning rather than the final version.</p><p>Building the system revealed several opportunities for extending the architecture while preserving the same design principles of specialization, structured contracts, deterministic execution, and observability.</p><p>One area I’m particularly interested in is expanding the team’s reasoning capabilities.</p><p>Today, the Model Evaluation Agent recommends which model should move forward based on validation and cross-validation results. Future versions could reason about explainability, fairness, inference latency, model complexity, or operational cost, turning model selection into a true multi-objective optimization problem.</p><p>Another promising direction is replacing exhaustive Grid Search with more efficient optimization techniques such as Bayesian Optimization or Optuna. Rather than evaluating every parameter combination, the team could intelligently explore the search space while reducing computational cost.</p><p>Feature engineering is another area with significant room for growth. The current workflow relies on the Feature Engineering Agent to propose transformations, but future versions could incorporate automated feature generation, feature selection, and explainability tools such as SHAP to better understand why specific models outperform others.</p><p>Another natural evolution of this architecture is closing the engineering loop completely. Today, every experiment is recorded together with its reasoning, evaluation, and outcome. Future versions could allow the agents themselves to analyze that history, formulate new hypotheses, propose the next experiment, execute it, and evaluate the results autonomously. At that point, the system would move beyond executing machine learning workflows and begin participating in its own continuous improvement.</p><p>Because every experiment is stored together with its reasoning, evaluation, and outcome, the experiment history could become a knowledge base that future agents consult before making recommendations. Instead of starting from scratch for every competition, the team could retrieve similar experiments, compare previous architectural decisions, and adapt successful strategies to new datasets.</p><p>The current workflow is also largely sequential. Many stages could eventually execute in parallel. Multiple feature engineering strategies, candidate models, or hyperparameter searches could be explored simultaneously before the Model Evaluation Agent compares the results and selects the best-performing approach.</p><p>Finally, I believe this architecture extends well beyond Kaggle competitions.</p><p>The same pattern of specialized reasoning agents, deterministic executors, structured contracts, observability, and experiment tracking could be applied to recommendation systems, forecasting, anomaly detection, MLOps pipelines, scientific computing, or even autonomous software engineering workflows.</p><p>For me, that’s the most exciting outcome of this project. The goal was never to build a better Titanic classifier. The goal was to explore how autonomous engineering teams can be designed, measured, and continuously improved. The Titanic competition simply became the laboratory where those ideas could be tested.</p><p>As Agentic AI continues to evolve, I suspect the most capable systems won’t be built around a single, increasingly powerful model. They’ll be built from collections of specialized components that collaborate through well-defined interfaces, continuously learn from objective feedback, and combine the strengths of language models with the reliability of traditional software engineering.</p><p>That’s the direction I’m excited to continue exploring.</p><h3>Conclusion</h3><p>This project started with a simple question:</p><p><strong>What if, instead of building an Agentic AI team that performs tasks with subjective results, we built an entire AI engineering team?</strong></p><p>Over the course of four articles, that question evolved from a research prototype into an autonomous data science system capable of solving end-to-end machine learning problems.</p><p>Along the way, I learned that the most important improvements didn’t come from writing longer prompts or switching to larger models.</p><p>They came from applying timeless software engineering principles to Agentic AI:</p><ul><li>Specialization over generalization</li><li>Explicit contracts instead of implicit assumptions</li><li>Deterministic execution instead of probabilistic computation</li><li>Observability instead of black-box workflows</li><li>Objective feedback instead of subjective evaluation</li></ul><p>Those principles allowed the system to become progressively more capable while remaining understandable, reproducible, and easy to extend.</p><p>Although the current implementation focuses on Kaggle competitions, I believe the underlying architecture is much more broadly applicable.</p><p>As AI systems become increasingly autonomous, I expect we’ll spend less time asking, <em>“How do I build a better prompt?”</em> and more time asking, <em>“How do I engineer better AI systems?”</em></p><p>For me, that’s the real purpose of this project. The Kaggle competition wasn’t the destination. It was the proving ground. The real objective was to explore how autonomous engineering teams can reason, collaborate, execute, measure their own performance, and continuously improve through objective feedback.</p><p>I believe we’re only beginning to discover what those systems will look like.</p><h3>Repository</h3><p>If you’d like to explore the implementation, the complete notebook and source code are available on GitHub:</p><p><strong>GitHub:</strong> <a href="https://github.com/icarovazquez/agentic-ai-data-science-team">https://github.com/icarovazquez/agentic-ai-data-science-team</a></p><p>The notebook includes the complete workflow, from problem framing through experiment tracking, observability, and Kaggle submission.</p><h3>Previous Articles in This Series</h3><p>This project builds on ideas developed throughout my earlier articles:</p><ul><li><a href="https://medium.com/@icaro_vazquez/agentic-ai-a-quick-overview-an-example-3aa538926e53"><strong>Agentic AI: A Quick Overview &amp; An Example</strong></a> — introducing specialized AI teams.</li><li><a href="https://medium.com/@icaro_vazquez/agentic-ai-a-more-generic-and-more-useful-example-9140b73027d4"><strong>Agentic AI: A More Generic (and more Useful?) Example</strong></a> — generalizing the task of a specialized Agentic AI Research Team</li><li><a href="https://medium.com/@icaro_vazquez/what-observability-revealed-about-my-agentic-ai-team-15e442e6a6d7"><strong>What Observability Revealed About My Agentic AI Team</strong></a> — using Langfuse to understand how agents collaborate.</li><li><a href="https://medium.com/@icaro_vazquez/improving-my-agentic-ai-team-how-better-contracts-structured-outputs-and-evaluations-improved-6dc0651cb287"><strong>Improving My Agentic AI Team: How Better Contracts, Structured Outputs and Evaluations Improved Performance</strong> </a>— evolving the architecture through structured communication and evaluation.</li></ul><p>Each article represents another step in the evolution of the architecture.</p><h3>I’d Love Your Feedback</h3><p>If you’re building agentic systems, autonomous machine learning pipelines, or AI engineering platforms, I’d love to hear how you’re approaching problems like specialization, observability, structured contracts, and experiment tracking.</p><p>One of the most exciting aspects of this field is that we’re all still discovering what good architectural patterns look like. I hope sharing this project contributes a few useful ideas to that conversation.</p><p>If you have suggestions, questions, or ideas for where this architecture should evolve next, feel free to reach out or open an issue on GitHub.</p><p>The next generation of AI systems won’t be individual agents. They’ll be engineering teams.</p><p><strong>P.S.</strong> The next evolution of this project will focus on giving the team memory — allowing agents to learn from previous experiments and use that knowledge to improve future machine learning workflows.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c6d38ed0f9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving My Agentic AI Team: How Better Contracts, Structured Outputs, and Evaluations Improved…]]></title>
            <link>https://medium.com/@icaro_vazquez/improving-my-agentic-ai-team-how-better-contracts-structured-outputs-and-evaluations-improved-6dc0651cb287?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dc0651cb287</guid>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Sat, 06 Jun 2026 01:10:25 GMT</pubDate>
            <atom:updated>2026-06-06T01:29:08.933Z</atom:updated>
            <content:encoded><![CDATA[<h3>Improving My Agentic AI Team: How Better Contracts, Structured Outputs, and Evaluations Improved Reliability</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4eWWCE40wHmsml6QKX8-Q.png" /></figure><h3>Introduction</h3><p>In the previous article in this series, I added observability to my Agentic AI team using Langfuse.</p><p>The traces revealed something surprising.</p><p>Most of the major failures were not model failures.</p><p>The Editor Agent bug was a contract failure.</p><p>Token explosion was a context management failure.</p><p>Off-topic reports were orchestration failures.</p><p>Observability helped me identify the problems.</p><p>The next challenge was fixing them.</p><p>This article documents the architectural improvements I made to transform the workflow from a collection of cooperating agents into a more reliable and predictable research platform.</p><p>Along the way, I learned an important lesson:</p><p><strong>Improving agentic systems is often less about selecting better models and more about designing better systems around those models.</strong></p><h3>The Problems Observability Exposed</h3><p>Before discussing solutions, it’s worth reviewing the problems observability uncovered.</p><p>The most important issues were:</p><ul><li>The Editor Agent producing critiques instead of final reports.</li><li>Reports that drifted away from the original topic.</li><li>Token consumption increased dramatically as context accumulated.</li><li>Agent implementation details leaking into user-facing outputs.</li><li>Report quality varied across runs.</li></ul><p>At first glance, these issues appeared unrelated.</p><p>In reality, they were symptoms of a deeper architectural problem.</p><p>The agents could communicate with one another, but the workflow lacked explicit contracts, validation mechanisms, context management controls, and meaningful evaluation criteria.</p><p>Observability made these weaknesses visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rGAqRKma6v-x7zgSa4op1w.png" /><figcaption><strong>Figure 1. Langfuse showing the execution flow of the Agentic AI Research Team.</strong></figcaption></figure><p>The remainder of this article focuses on the changes I made to address them.</p><h3>Structured Research Records</h3><p>One of the very first architectural weaknesses exposed by observability was the way the agents communicated with one another.</p><p>Originally, the Research Agent returned a prose summary of its findings. The output often looked reasonable when viewed in isolation. It contained key findings, supporting evidence, statistics, and sources.</p><p>The problem appeared when downstream agents consumed that output.</p><p>The Technical Writer was forced to interpret a large block of prose and determine which information represented evidence, which information represented conclusions, and which information represented uncertainty. The Editor Agent then received another large block of prose and attempted to improve it further.</p><p>Every handoff required interpretation.</p><p>Those interpretations introduced ambiguity.</p><p>In some runs, the Technical Writer Agent treated the Research Agent’s findings as source material that needed to be analyzed. In other runs, the Editor Agent treated the Technical Writer’s draft as a document to critique rather than a report to finalize.</p><p>The result was a persistent but subtle failure.</p><p>Reports occasionally contained phrases such as:</p><ul><li>“The prior work suggests…”</li><li>“The source material establishes…”</li><li>“The analysis above indicates…”</li></ul><p>These phrases were technically correct descriptions of what the agents had received, but they were inappropriate for a final report intended for the end user.</p><p>The report was exposing the internal implementation details of the workflow.</p><p>Observability made the problem visible.</p><p>After reviewing multiple traces, I realized the root cause was not the models themselves. The problem was the contract between agents.</p><p>The Research Agent was returning prose.</p><p>The downstream agents were forced to infer structure that was never explicitly defined.</p><p>To eliminate that ambiguity, I replaced the prose handoff with a structured research record.</p><p>Instead of generating narrative text, the Research Agent now returns a dictionary with explicit fields:</p><pre>{<br>  &quot;task&quot;: &quot;&quot;,<br>  &quot;key_findings&quot;:[],<br>  &quot;statistics&quot;:[],<br>  &quot;sources&quot;:[],<br>  &quot;risks_or_uncertainties&quot;:[],<br>  &quot;recommended_claims&quot;:[]<br>}</pre><p>Each field has a specific purpose:</p><ul><li>Key findings capture the most important conclusions.</li><li>Statistics contain quantitative evidence.</li><li>Sources document where the information originated.</li><li>Risks and uncertainties preserve areas where evidence is incomplete or debated.</li><li>Recommended claims provide candidate arguments for inclusion in the final report.</li></ul><p>Figure 2 below shows a partial research record generated by a run of the Agentic AI team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqEp6IfumH8cgow8dAE8DQ.png" /><figcaption><strong>Figure 2. Example of the Structured Research Record used by the Agentic AI team.</strong></figcaption></figure><p>This change fundamentally altered the relationship between agents:</p><ul><li>The Research Agent became responsible for gathering evidence.</li><li>The Technical Writer became responsible for synthesizing that evidence into a coherent narrative.</li><li>The Editor became responsible for refining the narrative rather than interpreting raw research.</li></ul><p>In other words, each agent received structured information instead of a document requiring interpretation.</p><p>The impact was immediate.</p><p>The report leakage that had repeatedly appeared during earlier experiments disappeared across multiple test topics, including historical analysis, financial analysis, and forecasting exercises.</p><p>More importantly, the workflow became easier to reason about. Each agent now consumed and produced well-defined artifacts.</p><p>The communication channel between agents had evolved from prose to an explicit contract.</p><p>To validate the contract, I ran the updated workflow across multiple topics, including:</p><ul><li>Why did the Soviet Union collapse?</li><li>What caused the 2008 financial crisis?</li><li>Could OpenAI become financially distressed within five years of a public offering?</li><li>Will AI create more jobs than it destroys over the next decade?</li></ul><p>In each case, the leakage that had previously appeared in the reports disappeared completely. More importantly, the reports began reading like reports rather than critiques of reports.</p><p>This was the first indication that improving the interfaces between agents could have a larger impact than changing the underlying models themselves.</p><h3>Agent Systems Fail at Interfaces</h3><p>The structured research record eliminated a major source of ambiguity, but the report leakage did not disappear immediately.</p><p>Even after updating the Research Agent and improving the contract between the Technical Writer Agent and the Editor Agent, I continued to see phrases such as:</p><ul><li>“The prior work suggests…”</li><li>“The source material establishes…”</li><li>“The analysis above estimates…”</li></ul><p>Figure 3 below shows the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uasgAAEP-Za7ydqFbIg51Q.png" /><figcaption><strong>Figure 3. Example of Implementation Details leaking into a user-facing report.</strong></figcaption></figure><p>At this point, it was becoming difficult to explain this behavior:</p><ul><li>The Research Agent was returning the research record instead of returning prose.</li><li>The Technical Writer was explicitly instructed to write as the report&#39;s author rather than as a reviewer.</li><li>The Editor was explicitly instructed to return the final report rather than an editorial critique.</li></ul><p>Yet, the leakage persisted.</p><p>This is where observability proved valuable once again.</p><p>Instead of treating the problem as a model failure, I began tracing the information passed between the agents and reviewing the prompts generated by the orchestration layer.</p><p>Eventually, I found the culprit. The issue was not inside any of the agents mentioned.</p><p>It was inside the Project Manager Agent.</p><p>As part of the task routing process, the Project Manager had an enriched task containing the new assignment and the output generated by previous agents. The prompt included the following instruction:</p><pre>Use the prior work as source material. Do not ignore it</pre><p>That sentence seemed harmless. In fact, it sounded like a reasonable thing to say to an agent that is part of a collaborative workflow.</p><p>Unfortunately, it was also introducing the exact language I was trying to eliminate.</p><p>The Project Manager was effectively telling downstream agents that they were reviewing source material rather than producing a final report for the end user. The agents responded accordingly.</p><p>The language embedded in the orchestration prompt began appearing in the final report. This was an important realization.</p><p>When building agentic systems, it is very easy to focus on individual agent prompts exclusively. However, the orchestration layer is also part of the system’s behavior.</p><p>Task routing prompts, context construction logic, memory management mechanisms, and workflow instructions can influence outputs just as strongly as the prompts assigned to individual agents.</p><p>The fix was straightforward. I rewrote the orchestration prompt so that it provided context without describing previous outputs as “prior work” or “source material”. The agents were encouraged to use the available information while writing directly to the end user.</p><p>After making this change, the leakage disappeared consistently across historical, financial, and forecasting topics.</p><p>Even more important, the reports began reading like final deliverables instead of reviews of intermediate artifacts.</p><p>This experience reinforced a lesson that became increasingly clear throughout the project:</p><p><strong>Agentic Systems often fail at the interfaces between agents.</strong></p><p>The most important bugs are not always inside the models themselves. Sometimes, they emerge from the way information is passed from one component of the system to another.</p><h3>Validation Layers and Leakage Detection</h3><p>Eliminating leakage through better agent contracts was a significant improvement, but it raised an important question:</p><p>How could I know for sure the problem was actually solved?</p><p>Manually reviewing each report was not scalable. As the number of test runs increased, it became increasingly impractical to inspect thousands of words of generated text looking for subtle signs of leakage.</p><p>I needed an automated validation mechanism.</p><p>This led me to introduce one of the simplest — and most useful — components of the workflow: a leakage detector.</p><p>The purpose of the detector was simple. After a report was generated, it scanned the output for phrases that should not appear in the final deliverable. Examples included:</p><ul><li>prior work</li><li>prior analysis</li><li>source material</li><li>research notes</li><li>previous draft</li><li>the analysis above</li></ul><p>These phrases are not necessarily incorrect. They are perfectly reasonable terms for agents communicating with each other.</p><p>However, they are implementation details. An end user should never know a report was assembled by multiple agents passing information between stages of a workflow.</p><p>The detector allowed me to automatically identify reports where those implementation details leaked into the final output.</p><p>The implementation itself was intentionally simple. Rather than building a sophisticated classifier, I started with a rule-based approach that searched for known leakage phrases, as shown in the function below.</p><pre>def detect_leakage(report_text: str):<br><br>    forbidden_phrases = [<br>        &quot;source material&quot;,<br>        &quot;prior work&quot;,<br>        &quot;prior analysis&quot;,<br>        &quot;research notes&quot;,<br>        &quot;provided material&quot;,<br>        &quot;previous draft&quot;,<br>        &quot;the analysis above&quot;,<br>        &quot;the report above&quot;,<br>    ]<br><br>    report_lower = report_text.lower()<br><br>    found = []<br><br>    for phrase in forbidden_phrases:<br>        if phrase in report_lower:<br>            found.append(phrase)<br><br>    return found</pre><p>The goal was not perfect detection. The goal was to create a fast feedback loop.</p><p>If a report contained one of these phrases, I immediately knew that some component of the workflow was violating the contract I had established.</p><p>This simple validation layer proved surprisingly effective.</p><p>Before introducing structured research records and fixing the Project Manager prompt, the detector frequently identified leakage in generated reports.</p><p>After the architectural changes were implemented, the leakage score consistently dropped to zero across multiple test topics.</p><p>The detector transformed leakage from a subjective observation into a measurable metric.</p><p>More importantly, it reinforced an engineering principle that applies far beyond agentic systems:</p><p><strong>Do not trust outputs simply because they look correct. Validate them.</strong></p><p>As the workflow continued to evolve, I found myself increasingly adopting the same mindset used when building traditional software systems. Generating content was only one part of the problem.</p><p>Reliable systems also require validation, monitoring, testing, and quality controls</p><h3>Improving the Judge</h3><p>By this point, the reports looked significantly better. Leakage disappeared. Agent responsibilities were clearer. The workflow was producing coherent reports across historical, financial, and forecasting topics.</p><p>However, another problem emerged. The evaluation system was not improving alongside the workflow.</p><p>I had implemented a simple LLM-as-a-judge evaluator that scored reports on relevance, competence, and clarity.</p><p>At first, the evaluator seemed useful. Over time, a strange pattern appeared. Almost every report received the same score.</p><p>Historical reports received a score of 0.8. Forecasting reports received a score of 0.8. Reports that I considered significantly better than earlier versions still received a score of 0.8.</p><p>The evaluator was not helping me distinguish between average reports and good reports.</p><p>After investigating the issue, I discovered a major limitation of my implementation. The evaluator was only examining the first 1,000 characters of each report. This meant the evaluator was only examining the title, executive summary, and opening paragraphs rather than the complete analysis.</p><p>The evaluator simply didn’t have enough information to accurately assess report quality.</p><p>To address this issue, I redesigned the evaluator using a more detailed rubric. Instead of scoring reports using only relevance, completeness, and clarity, the new evaluator considered:</p><ul><li>Relevance</li><li>Completeness</li><li>Analytical depth</li><li>Evidence quality</li><li>Structure and clarity</li><li>Uncertainty handling</li><li>Executive usefulness</li></ul><p>I also gave the evaluator a much larger version of the report to review.</p><p>Most importantly, it was required to explain its reasoning by identifying strengths and weaknesses rather than just returning a numeric score. Figure 4 below shows an example of the new evaluator output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iPgPrdWFkO73vtouwkmFKg.png" /><figcaption><strong>Figure 4. An example of the new LLM-as-a-Judge Evaluator. This version uses a more detailed rubric and examines a much larger version of the report.</strong></figcaption></figure><p>The difference was immediate. Reports that previously clustered around a 0.8 score began receiving more differentiated evaluations.</p><p>The evaluator had become a more useful measurement instrument.</p><p>This experience reinforced another lesson from the project: As the system improves, the measurement system often becomes the next bottleneck.</p><p>Improving the agents revealed weaknesses in the evaluator that had been previously hidden by larger architectural problems. In other words, improving the workflow required improving the way the workflow was measured.</p><h3>Results Across Multiple Topics</h3><p>Architectural improvements are only useful if they generalize beyond a single example.</p><p>The improvements described in this article were implemented incrementally rather than all at once. Figure 5 below summarizes the major milestones in the evolution of the workflow during this phase of the project.</p><p>The progression is worth noting because each improvement built on the previous one. Observability exposed the weaknesses, structured research records improved the contracts between agents, and the redesigned evaluator provided a more reliable way to measure the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B32EblNUeYGZ5AxBrr9e-w.png" /><figcaption><strong>Figure 5. Git commit history showing the evolution of the Agentic AI Research Team.</strong></figcaption></figure><p>To validate the workflow, I tested the updated system across a diverse set of topics spanning history, finance, technology, and forecasting. The test topics included:</p><ul><li>Why did the Soviet Union collapse?</li><li>What caused the 2008 financial crisis?</li><li>Could OpenAI become financially distressed within 5 years of a public offering?</li><li>Will AI create more jobs than it destroys over the next decade?</li></ul><p>The diversity of these topics was intentional. I wanted to determine whether the improvements generalized across domains rather than improving performance on a single benchmark problem.</p><p>The goal was not to verify the correctness of every conclusion. Instead, I wanted to determine if the Agentic AI team was consistently producing coherent, well-structured reports while avoiding the failure modes that observability had previously exposed.</p><p>The results were encouraging. Across all test topics:</p><ul><li>Report leakage disappeared.</li><li>The Editor Agent consistently produced final deliverables rather than critiques.</li><li>Reports remained focused on the original question.</li><li>Agent responsibilities remained clearly separated.</li><li>Evaluation scores became more differentiated.</li></ul><p>Although the reports were not perfect, the failure modes became more predictable and easier to diagnose. This was an important improvement. The goal was never to build a flawless system; the goal was to build a system whose behavior could be understood, measured, and improved over time.</p><p>The AI employment report provided one of the strongest examples. The improved evaluator assigned it a score of 0.95 and identified both strengths and weaknesses rather than simply returning a generic score.</p><p>While the system is still far from perfect, the improvements demonstrated that better contracts, validation mechanisms, and evaluation processes can significantly improve the reliability of multi-agent workflows.</p><p>Perhaps most importantly, the changes generalized across very different domains. The same architecture successfully handled historical analysis, financial analysis, strategic forecasting, and technology trends without requiring topic-specific modifications.</p><h3>Lessons Learned</h3><p>The main lessons I learned throughout this project are:</p><p><strong>Lesson 1: Observability Reveals Architectural Problems.</strong></p><p>Before adding observability, many of the workflow’s problems appeared random. Reports were sometimes incomplete, occasionally off-topic, and often inconsistent in quality. The traces revealed that these were not isolated problems but symptoms of deeper architectural issues. Observability transformed the workflow from a black box to into an understandable system and provided the evidence needed to make meaningful improvements.</p><p><strong>Lesson 2: Agent Systems Fail At Interfaces.</strong></p><p>One of the most important discoveries was that many failures did not originate inside individual agents. Instead, they emerged at the boundaries between agents. Ambiguous contracts, conflicting instructions, and poorly structured content propagation created failure modes that were difficult to diagnose without tracing. Improving the interfaces between agents often produced greater gains than modifying the agents themselves.</p><p><strong>Lesson 3: Structured Outputs Are Contracts.</strong></p><p>Introducing structured research records fundamentally changed the workflow. Rather than forcing downstream agents to interpret large blocks of prose, each agent received a well-defined artifact with a clear purpose. This reduced ambiguity, eliminated report leakage, and made the system easier to reason about. The lesson extends beyond this project: whenever possible, agent communications should rely on explicit contracts rather than inferred meaning.</p><p><strong>Lesson 4. Evaluation Systems Become Bottlenecks</strong></p><p>As the quality of the reports improved, weaknesses in the evaluation framework became more apparent. The original evaluator could not distinguish between average and excellent reports because it lacked sufficient context and used an overly simplistic rubric. Improving the workflow required improving the measurement system. Better outputs demand better evaluation systems.</p><p><strong>Lesson 5: Restart your Google Colab Runtime After Prompt Changes.</strong></p><p>This lesson is less glamorous than the others, but it proved surprisingly important. During debugging, I also discovered that prompt changes in a notebook environment can create misleading results if the stale prompts remain loaded in memory. Several times I modified prompts and forgot to restart the runtime, causing the workflow to behave differently from what I expected. This is a mundane engineering issue, but one that becomes increasingly important as agentic systems grow in complexity. Sometimes, the bug is not in the workflow at all — it is in the development environment.</p><h3>Conclusion</h3><p>Throughout this project, I was repeatedly reminded that these systems are far more accessible than many people assume. The workflow described here was not built by a large AI engineering organization. It was built incrementally through experimentation, debugging, measurement, and iteration.</p><p>My hope is that readers — whether they are students, product managers, analysts, entrepreneurs, or simply curious learners — finish this article with a different perspective on AI. These systems are not magic. They are systems that can be understood, improved, and built by ordinary people willing to learn.</p><p>The barrier to entry is lower than ever. The most important requirement is not access to advanced technology or having a technical degree. It is curiosity and a willingness to learn.</p><p>Many of the concepts explained here — agents, observability, evaluations, structured outputs, and workflow orchestration — can sound intimidating at first. In practice, they are often simpler than they appear:</p><ul><li>An agent is simply a model with a defined role.</li><li>A workflow is simply a sequence of tasks.</li><li>Observability is simply a way to understand what the system is doing.</li></ul><p>The most important lesson is that building useful AI systems is becoming less about advanced Artificial Intelligence and more about thoughtful system design.</p><p>That is good news.</p><p>It means that you, students, future product managers, analysts, entrepreneurs, and professionals from different backgrounds can begin experimenting with these technologies today.</p><p>You do not need to work at an AI company to build AI systems. You do not need to work at a Silicon Valley company to build Agentic AI workflows.</p><p>You simply need curiosity, a willingness to experiment, and a problem worth solving.</p><h3>What’s Next?</h3><p>So far, this series has focused on building, observing, and improving an Agentic AI research team.</p><p>The next challenge is determining whether these same architectural principles can be applied to more objective and measurable problems.</p><p>Research reports are useful, but evaluating their quality still involves some degree of human judgment. What happens when the agents are given a problem with a clear quantitative definition of success?</p><p>In the next article, I plan to explore how an Agentic AI team can tackle a real-world machine learning problem and evaluate whether agents can meaningfully improve performance through structured collaboration, evaluation, and iteration.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dc0651cb287" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Observability Revealed about My Agentic AI Team]]></title>
            <link>https://medium.com/@icaro_vazquez/what-observability-revealed-about-my-agentic-ai-team-15e442e6a6d7?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/15e442e6a6d7</guid>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Wed, 03 Jun 2026 06:43:36 GMT</pubDate>
            <atom:updated>2026-06-03T06:48:40.038Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VYqE3HKU7Gr6eh1WaJAXyw.png" /></figure><h3>Introduction</h3><p>This is the third installment in an ongoing series documenting the evolution of an Agentic AI research team that I have been building using Python.</p><p>In the <a href="https://medium.com/@icaro_vazquez/agentic-ai-a-quick-overview-an-example-3aa538926e53">first article</a>, I introduced the initial concept: a collection of specialized AI agents that could collaborate to conduct research and generate reports. The goal was to move beyond simple prompting and explore what happens when multiple agents work together toward a common objective.</p><p>In the <a href="https://medium.com/@icaro_vazquez/agentic-ai-a-more-generic-and-more-useful-example-9140b73027d4">second article</a>, I generalized the architecture so the team could tackle a much broader range of research topics. Along the way, I added additional agents, and workflow orchestration logic, making the system significantly more capable — but also more complex.</p><p>This article focuses on the next challenge that emerged: observability.</p><p>As the number of agents, prompts, and interactions increased, the workflow became increasingly difficult to understand and debug. I could see the final output, but I had very little visibility into how that output was generated.</p><p>To solve the problem, I integrated Langfuse into the workflow and began tracing every agent execution, LLM call, prompt, response, and token consumed.</p><p>What followed was one of the most educational parts of the project. Observability not only helped me to debug the system — it fundamentally changed how I thought about designing Agentic AI architectures.</p><h3>Series Roadmap</h3><p>The evolution of the project can be summarized in four major phases, shown in Figure 1 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oUPOphhIhmhWXfmqKGDerQ.png" /><figcaption><strong>Figure 1. Agentic AI Team Phases</strong></figcaption></figure><h3>The Original Vision</h3><p>Like many engineers and product managers experimenting with Agentic AI to increase our productivity, I started with a simple goal: create a team of AI agents that could conduct research for me and produce an initial, polished report without any human intervention.</p><p>The architecture was straightforward:</p><ul><li>A Corporate Strategy Manager would create the research plan</li><li>A Project Manager Agent would assign the research plan tasks to the other members of the research team</li><li>A Research Agent would gather information using tools like the web, arXiv, and Wikipedia</li><li>A Technical Writer Agent would summarize the research findings in a coherent report</li><li>An Editor Agent would review the Technical Writer’s output, criticize it, make adjustments, and produce a polished final report</li><li>Finally, a Marketing Manager agent would synthesize the report’s findings in a PowerPoint presentation</li></ul><p>Together, these agents formed a multi-agent research workflow that could transform any research topic into both a written report and an executive presentation.</p><p>At a high level, the workflow looked like this:</p><p>Corporate Strategy Manager -&gt; Research Agent -&gt; Technical Writer Agent -&gt; Editor Agent -&gt; Marketing Manager Agent.</p><p>The entire system was implemented in a Google Colab Notebook using Python and Andrew Ng’s <a href="https://github.com/andrewyng/aisuite">aisuite library</a>, which provided a common interface to different LLM providers.</p><p>Initially, the system appeared to work well. Give my team a research topic, and the agents collaborated to generate a reasonably structured report. Early runs were promising, and the architecture felt surprisingly simple.</p><p>As often happens with AI systems, the initial euphoria and simplicity were misleading.</p><h3>The System Worked… Until It Didn’t</h3><p>As I started experimenting with more complex research topics that lend themselves to longer reports, not-so-subtle problems started to emerge.</p><p>Some reports ended abruptly; others had sections that felt incomplete. In several cases, the Editor agent returned an editorial review instead of the rewritten report it was supposed to produce. In other instances, the subject of the report produced was not the correct one; it was off-topic; or, if we are generous, the subject was a tangential one and not the main one.</p><p>At first, these issues seemed unrelated.</p><p>One report was missing the conclusion. Another would generate a critique and not the deliverable we wanted. Occasionally, the workflow would hit the LLM subscription API limit by consuming more tokens than expected.</p><p>The challenge was that I had zero visibility into what was actually happening inside the system.</p><p>From the outside, all I could see was the final output. I had a few log messages that I added in some of the Notebook’s cells, but it was nothing deep. If the report was incomplete, I didn’t know if it was the Research Agent who didn’t gather the appropriate information, whether the Technical Writer had omitted important details, or whether the Editor had simply stopped generating text because we hit a token limit.</p><p>Similarly, if we hit a limit, I would go to the LLM console (Claude, for example) and see that we hit the limit, but I had no way of knowing which agent(s) was responsible.</p><p>On paper, the workflow appeared linear. In reality, I suspected the system was behaving like a distributed network of reasoning steps whose internal behavior was invisible to me.</p><p>The more successful the system became, the harder it was to debug. Every new agent or every new capability added to an agent increased the number of moving parts and made failures harder to diagnose.</p><p>This is where observability became essential.</p><p>Rather than just adding more logs in whatever notebook cell I felt was needed, I decided to instrument the whole workflow using <a href="https://langfuse.com/">Langfuse</a>, so I could trace every agent execution, every LLM call, every prompt, and every token consumed during each run.</p><h3>Adding Observability</h3><p>At this point, I reached the familiar stage in the evolution of many software systems: the system was too complex to understand through log messages alone.</p><p>I could see the final report, but I could not see how the final report was produced.</p><p>I wanted answers to questions such as:</p><ul><li>Which agent was consuming the most tokens?</li><li>Is it always the same agent, or does it change?</li><li>How many LLM calls were being executed during a run?</li><li>Which agent was responsible for a failure?</li><li>Why were some runs slower than others?</li><li>Were agents actually using the context they were receiving?</li></ul><p>To answer these questions, I introduced Langfuse into the workflow. Adding Langfuse was not very complex. Basically, it meant adding a decorator in every function that I want to track, as shown in the code snippet below.</p><pre>@observe(name=&quot;llm_call&quot;, as_type=&#39;generation&#39;)<br>def llm_call(agent_name:str, messages: list, temperature: float = 1.0, tools: list= None) -&gt; str:<br><br>  &quot;&quot;&quot;<br>  observability wrapper that all agents will use when calling the llm<br>  logs model, input, output, and token usage to Langfuse<br>  &quot;&quot;&quot;</pre><p>Those decorators gave me:</p><ul><li>Tracing</li><li>Token Tracking</li><li>Evaluations</li><li>Execution timelines</li></ul><p>At the time, I evaluated several observability options. I ultimately selected Langfuse because it was specifically designed for LLM applications, and provided built-in support for traces, generations, evaluations, token tracing, and prompt inspection. Just as important, it was straightforward to integrate into a Python-based workflow running in Google Colab.</p><p>Rather than simply logging messages to the notebook, Langfuse allowed me to trace the execution of the whole workflow. Every agent invocation, every LLM call, every response, and every token consumed became visible.</p><p>The first thing I realized after opening the Langfuse timeline was that my mental model of the system was wrong. Before observability, I thought I had built a linear workflow (Corporate Strategy Manager -&gt; Research Agent -&gt; Technical Writer Agent -&gt; Editor Agent -&gt; Marketing Manager Agent). What I actually had built was a hierarchy of nested LLM calls, agent interactions, and context propagation mechanisms.</p><p>The Langfuse timeline, shown in Figure 1 below, revealed something more interesting.</p><p>The workflow was not the simple sequence of agents. It was an execution tree composed of nesting reasoning steps, LLM calls, and context propagation mechanisms.</p><p>Figure 2 shows a single execution of the workflow as captured by Langfuse</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hugs9Gt8w2247k-2E0w0nA.png" /><figcaption><strong>Figure 2. Langfuse Timeline of the Agentic AI Research Team</strong></figcaption></figure><p>For the first time, I could see what was executed, in what order it was executed, how long each step had taken, and how the various components of the system interacted with one another.</p><p>The timeline transformed the workflow from a black box into an observable system.</p><p>The traces were useful, but the real value came from the lessons hidden inside them.</p><h3>What Observability Revealed</h3><p>Adding Observability gave me visibility into the workflow, but the real value came from the architectural lessons that emerged from the traces.</p><p>The timeline, the token metrics, and execution traces revealed several issues that would have been difficult — or impossible — to identify from the output alone.</p><h3>Agent Failures Were Often Contract Failures Rather Than Model Failures</h3><p>This was my first indication that designing agentic systems is as much about designing interfaces between agents as it is about selecting models.</p><p>One of the most interesting surprises was the Editor Agent.</p><p>Its purpose was straightforward: take the Technical Writer’s output and produce a polished report.</p><p>However, some runs produced something entirely different. Instead of a revised report, the Editor generated a detailed editorial review containing findings, critiques, and recommendations—great information, but not the output I was expecting.</p><p>At first, I thought what a lot of us have thought when something like this happens: maybe it’s the model. That was not the problem; the problem was the contract between agents.</p><p>The Editor Agent was instructed to critique drafts, improve them, and produce a final report. At first glance, those instructions sound compatible, but they created ambiguity. In some runs, the editor interpreted its task as “perform an editorial review”. In others, it interpreted it as “rewrite the report”.</p><p>Observability helped to isolate the failure to a specific agent and a specific prompt rather than me treating it as a system-wide problem.</p><p>The fix was surprisingly simple: tighten the agent contract by modifying a prompt.</p><p>Instead of asking the editor to critique and improve the draft, I explicitly asked it to, in addition to the above tasks, act as the author of the final report and to return only the final deliverable.</p><p>This experience reinforced an important lesson: agent failures are often contract failures rather than model failures.</p><h3>The Workflow Was Growing More Expensive Than I Realized</h3><p>The Langfuse traces also revealed something that I had not been paying close attention to: token consumption.</p><p>One of the first operational realities of agentic systems is that context tends to grow over time.</p><p>My initial design assumed that every agent should receive the complete output generated by previous agents. While this approach maximized context, it also created an initially hidden cost: every handoff increased the size of future prompts.</p><p>As reports became longer, the amount of context passed by agents increased dramatically. Larger prompts required more tokens, which increased both cost and latency.</p><p>What appeared to be a simple report-generation workflow was actually consuming tens of thousands of tokens across multiple LLM calls.</p><p>In one run, my agents consumed more than 56,000 tokens:</p><ul><li>Input Tokens: 27,912</li><li>Output Tokens: 28,923</li><li>Total Tokens: 56,835</li></ul><p>Without observability, I was flying blind, and I would only have known that I had reached my API usage limit.</p><p>With observability, I could identify exactly where the tokens were consumed and which agents were responsible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vvjT6EezlhBvwGqMq1bjg.png" /><figcaption><strong>Figure 3. Langfuse generation view showing token consumption for the Editor Agent. Observability allowed me to attach token usage to individual agents rather than treating the workflow like an opaque API call</strong></figcaption></figure><p>This transformed token consumption from a vague operational concern to a trackable engineering metric. Figure 3 above shows a screenshot of a recent run of my Agentic AI team, concentrating on the Editor agent. As you can see, it shows the editor agent consumed 7,356 tokens.</p><h3>The Workflow Was Not Actually Linear</h3><p>Perhaps the most important lesson was that the workflow was not behaving in the way I imagined.</p><p>Before observability, I thought of the system as a series of agents passing information from one agent to the next one.</p><p>The workflow was not behaving like a traditional software pipeline. Instead, the timeline revealed something closer to a distributed cognitive workflow:</p><ul><li>Agent execution triggered LLM calls.</li><li>LLM calls generated outputs.</li><li>Outputs generated context.</li><li>Contents influenced future reasoning steps.</li><li>Evaluation logic examined results before the workflow continued.</li></ul><p>The resulting behavior was not a simple sequence of steps but a graph of interconnected reasoning processes.</p><p>This distinction fundamentally changed how I thought about agent design. Rather than viewing agents as prompts connected, I began viewing them as interacting cognitive components whose behavior emerges from the flow of information between them.</p><h3>Observability Is More Than Tracing</h3><p>As I became more comfortable with Langfuse, I realized that Agentic AI observability extends beyond tracing alone.</p><p>I also wanted to measure the quality of the output generated by the system. To accomplish this, I introduced a simple evaluation approach using an LLM-as-a-judge approach.</p><p>After each work execution, a separate model evaluated the final report and generated scores for:</p><ul><li>Report Quality</li><li>Tool Selection Correctness</li><li>Research Depth</li></ul><p>The results were automatically attached to the trace and became part of the observable execution record, as shown in Figure 4 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8T7Z9hY-w4z37X62JqQCwg.png" /><figcaption><strong>Figure 4. Langfuse view showing the 3 scores generated by the LLM-as-a-judge approach.</strong></figcaption></figure><p>This changed the way I thought about observability.</p><p>Traditional software observability focuses on metrics such as latency, throughput, and failures.</p><p>Agentic systems introduce an additional challenge: output quality.</p><p>A workflow can execute perfectly from a technical perspective and still produce a poor report. As noted above, this happened to me many a time.</p><p>By attaching evaluation scores directly to traces, I gained visibility into both operational behavior and output quality.</p><h3>How Observability Changed The Architecture</h3><p>Another big surprise is that observability did not simply help me debug the system.</p><p>It changed the system itself.</p><p>Once I could see traces, token usage, execution timelines, and evaluation scores, it became obvious that several assumptions underlying my original architecture were flawed.</p><h3>Centralizing LLM Calls</h3><p>One of the first changes I made was introducing a centralized llm_call() function.</p><p>Rather than allowing each agent to call the model independently, all LLM requests flowed through a common wrapper.</p><p>This provided a single location to:</p><ul><li>Track token consumption</li><li>Record model usage</li><li>Capture prompts and responses</li><li>Add Langfuse instrumentation</li><li>Implement future guardrails</li></ul><p>The result was a simpler architecture and significantly better visibility.</p><h3>Managing Context Growth</h3><p>Observability revealed that context growth was one of the largest drivers of token consumption.</p><p>Initially, every agent inherited the complete output generated by previous agents.</p><p>While this maximized context, it also caused prompts to grow continuously as the workflow progressed.</p><p>To address this issue, I introduced, as helper functions, context management mechanisms such as message trimming and running summaries that prevented prompt sizes from growing indefinitely.</p><h3>Improving Agent Contracts</h3><p>The Editor Agent failure highlighted the importance of explicit contracts between agents.</p><p>Research Agents gather evidence.</p><p>Technical Writers create drafts.</p><p>Editors produce final deliverables.</p><p>The more explicit the responsibility of each agent, the more predictable the behavior of the overall system.</p><h3>Adding Validation Layers</h3><p>Observability also encouraged me to introduce simple validation mechanisms.</p><p>One example was a leakage detector designed to identify phrases such as “source material” or “prior work” in final reports. This was in response to the editor agent being confused about its tasks.</p><p>Although simple, this validation layer helped detect situations where implementation details leaked into user-facing outputs.</p><p>After implementing all these changes, the reports became more complete, more consistent, less expensive to generate, and significantly easier to debug.</p><p>This marked an important shift in my thinking.</p><p>Building reliable agentic systems is not simply about generating content.</p><p>It is about generating, validating, measuring, and continuously improving the final output.</p><p>Looking back, the most surprising realization was that none of the major issues I discovered were model failures.</p><p>The Editor Agent bug was a contract failure.</p><p>The token explosion was a context management failure.</p><p>The off-topic reports were orchestration failures.</p><p>Observability helped me understand that building reliable agentic systems is often less about selecting a better model and more about improving the architecture surrounding the model.</p><h3>Lessons Learned</h3><p>Before adding observability, I viewed my agentic workflow as a collection of prompts connected together.</p><p>After adding observability, I began viewing it as a distributed system.</p><p>The difference is significant.</p><p>Distributed systems require monitoring, tracing, metrics, validation and debugging tools. Agentic systems are no different.</p><p>The most valuable lesson was not learning how many tokens my agents consumed or how long they took to execute, although those were great learnings.</p><p>The most valuable lesson was discovering how often my assumptions about the system were wrong.</p><p>Observability exposed hidden architectural flows, clarified agent responsibilities, revealed unexpected costs, and ultimately led to a more reliable design.</p><p>As agentic systems continue to grow in complexity and received increased usage, I believe observability will become a foundational capability rather than an optional enhancement. The costs of flying blind, both in the quality of the output and the costs of the system, are just too great to ignore.</p><p>We are moving beyond prompt engineering. We are engineering systems composed of collaborating AI agents.</p><p>And like every complex system before them, these systems require tracing, measurement, validation and observability if we expect them to be reliable.</p><p>Many of the improvements described in this article emerged directly from insights uncovered through observability.</p><p>In the next article, I will dive deeper into those changes, including context management, agent contracts, validation layers, structured outputs, leakage detection, and evaluation-driven development. Together, these techniques transformed the system from a collection of cooperating agents into a more reliable and predictable research platform.</p><p>Ultimately, Observability is not simply a debugging tool for agentic systems.</p><p>It is rapidly becoming a foundational engineering discipline.</p><p>As agentic architectures continue to evolve, the ability to trace, measure, validate, and improve their behavior will be as important as the models that power them.</p><h3>Resources</h3><ul><li>Source Code: <a href="https://github.com/icarovazquez/agentic-ai">GitHub Repository</a></li><li>Observability Platform: <a href="https://langfuse.com/">Langfuse</a></li><li>LLM Framework: <a href="https://github.com/andrewyng/aisuite">Andrew Ng’s AI Suite</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15e442e6a6d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI: A more generic (and more useful?) example]]></title>
            <link>https://medium.com/@icaro_vazquez/agentic-ai-a-more-generic-and-more-useful-example-9140b73027d4?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9140b73027d4</guid>
            <category><![CDATA[llms-utilization]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Thu, 11 Dec 2025 06:54:46 GMT</pubDate>
            <atom:updated>2025-12-11T06:54:46.287Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*6V3UMhH0rrvxgLRE4UGNSA.jpeg" /></figure><p>In a <a href="https://medium.com/@icaro_vazquez/agentic-ai-a-quick-overview-an-example-3aa538926e53">previous post</a>, we provided a high-level overview of what Agentic AI is and introduced an example of a team of AI agents collaborating to complete a task. I called this team the Corporate Strategy Team, and its task was to complete a hard-coded task that I assigned to it. In this case, the task was to produce a report that gave an opinion on the effect of the tariffs implemented by the Trump Administration. At the end of the day, the Corporate Strategy team produced a report in PDF format answering that question.</p><p>In this post, we evolve the Corporate Strategy team so it can work on any research topic suggested by you, the end user. The idea is that this team of AI agents can do research on pretty much any subject and produce a written report and a set of slides summarizing its findings. Even though I call it a Corporate Strategy Research Team, think of it as a team of agents that can do research on any topic and can be helpful in many situations. Some opportunities that come to mind are:</p><ul><li><em>Product Management:</em> You are a product manager who is writing a new PRD or writing stories for a new feature, and want to see if the competition has something similar and how it is received in the marketplace.</li><li><em>Corporate Strategy</em>: You are a Corporate Strategy team and are researching which companies you might acquire to complement your product portfolio</li><li><em>Market Analyst</em>: You are an analyst at places like Gartner and want/need to research a new market or a new company.</li><li><em>Job Search</em>: You are someone who is considering joining a new company and wants to research the company, its growth prospects, culture, and future outlook</li><li><em>News Reporter</em>: You are writing a story about a company and want to learn more about the company, its people, its products, and its future.</li><li><em>New Small Business</em>: You are thinking of starting a new small business in your town (a flower shop, a ladies fashion store, a nails salon, a dry cleaner, etc.) but want to figure out where would be the best location to put your business so you can maximize revenue. You also want to check the competition so you know what you are getting into.</li></ul><p>I can think of many others, but let’s keep this list short for now. I would love to hear in the comments how else you would use this team of AI Agents.</p><p>OK, now let’s take a look at the team.</p><h3>Corporate Strategy Team Members</h3><p>This team has the following members/agents:</p><ul><li>A Corporate Strategy Manager who generates the research plan and assigns owners to each task</li><li>A Research Assistant whose job is to find information about the research topic and write a summary of its findings</li><li>A Technical Writer who will create a research report</li><li>An Editor who will review the first version of the report, reflect on it, suggest improvements, and write a revised version incorporating the feedback</li><li>A Marketing Manager who creates slides from the final report generated by the other members of the team</li><li>A Project Manager who takes the research plan generated by the Corporate Strategy manager and makes sure the task owners execute their homework</li></ul><p>The deliverables, at the end of the day, are the research report in docx format and a presentation in ppt format. We changed these deliverables a bit from the ones in the last post. We removed the addition of graphics and added the automatic generation of slides since, in most cases, the research owner will be asked to present their findings.</p><p>OK, now that we know what the team is going to do and who the members are, let’s take a look at the code.</p><h3>Setting up the Environment</h3><p>As in the previous post, the code is located in a Google Colab notebook. In a future post, we will deploy it as an API to complete the development cycle. For now, it’s just executable in a notebook.</p><p>Let’s start with installing the needed libs:</p><pre>#install needed libs<br>!pip install aisuite<br>!pip install anthropic<br>!pip install tavily-python<br>!pip install openai<br>!pip install pypandoc<br>!pip install markdownify<br>!pip install tiktoken</pre><p>We are using the following libs:</p><ul><li>aisuite: Andrew Ng’s team created this library to facilitate access to different LLM providers</li><li>Antrophic: We are using Claude as the main LLM</li><li>Tavily-python: Tavily is a library that allows us to do web searches</li><li><a href="https://pypi.org/project/pypandoc/">pypandoc</a>: We use this library to create the docx version of the report</li><li>markdownify: Gets rid of the markdown in the report. Need to get rid of the markdown because the slides api we are using does not accept markdown docs</li><li>tiktoken: Enables us to reduce the size of the prompt sent to the api that will generate our slides</li></ul><p>Now, we import all the necessary components, obtain API keys, initialize clients, and set up the tools that will be made available to the LLM. Notice we are using an additional api; the <a href="https://slidesgpt.com/">slidesgpt</a> api. All of this is shown below.</p><pre>#import all the needed libs<br>import base64<br>import json<br>import os<br>import re<br>from datetime import datetime<br>from google.colab import userdata<br>from io import BytesIO<br>import pprint<br>import requests<br>import textwrap<br>import ast<br>import pypandoc #to create the docx version of the final report<br><br># 3rd Party<br>from IPython.display import display, Image as ImageDisplay, IFrame, Markdown<br>import aisuite as ai<br>import urllib, urllib.request<br>import urllib.parse # Import urllib.parse<br>import openai<br>from tavily import TavilyClient<br>from google.colab import files<br>from markdownify import markdownify<br>import tiktoken<br><br>os.environ[&#39;ANTHROPIC_API_KEY&#39;] = userdata.get(&#39;ANTHROPIC_API_KEY&#39;)<br>os.environ[&#39;TAVILY_API_KEY&#39;] = userdata.get(&#39;TAVILY_API_KEY&#39;)<br>os.environ[&#39;OPENAI_API_KEY&#39;] = userdata.get(&#39;OPENAI_API_KEY&#39;).strip()<br>os.environ[&#39;SLIDESGPT_API_KEY&#39;] = userdata.get(&#39;SLIDESGPT_API_KEY&#39;)<br><br><br>#initialize ai client<br>Client = ai.Client()<br><br>#Initializing research tools (tavily, arvix &amp; wikipedia)<br><br>#initialize tavily client<br>tavily_client = TavilyClient(api_key=os.environ[&#39;TAVILY_API_KEY&#39;])<br><br><br>#set arvix sample search term<br>arvix_search_term = &#39;impact of tariffs in the economy&#39;<br>arvix_url = f&#39;http://export.arxiv.org/api/query?search_query={urllib.parse.quote_plus(arvix_search_term)}&amp;start=0&amp;max_results=1&#39; # Encode the search term<br><br><br>#set wikipedia url<br>wikipedia_search_url= &#39;https://api.wikimedia.org/core/v1/wikipedia/en/search/page&#39;<br>headers = {<br>  &#39;User-Agent&#39;: &#39;MyWikipediaApp/1.0 (icarovazquez@gmail.com)&#39;<br>}<br><br># We are going to use claude for this team<br>model=&quot;anthropic:claude-3-7-sonnet-20250219&quot;<br><br>#slidesGPT API endpoint<br>slidesgpt_api_endpoint = &quot;https://api.slidesgpt.com/v1/presentations/generate&quot;</pre><p>As in our last post, the tools available for the research agent to use are: Tavily for web searches, arxiv for searching academic papers, and Wikipedia.</p><p>The notebook has a cell for running sample queries for each tool to make sure they are working correctly, but I’m not going to show it here; you can look at it in the previous post.</p><p>What we will show is the functions for each tool, which are shown below.</p><pre># Tavily web search function<br>def tavily_search(web_query: str):<br>    &quot;&quot;&quot;Search the web using Tavily.&quot;&quot;&quot;<br>    search_response = tavily_client.search(web_query)<br>    # pprint.pprint(search_response) # Remove this to avoid printing raw response<br>    # data = search_response.json() # Tavily response is already a dictionary<br>    return search_response<br><br># Arxiv search function<br>def arvix_search(arvix_query: str):<br>    &quot;&quot;&quot;Search academic papers with Arvix &quot;&quot;&quot;<br>    arvix_url = f&#39;http://export.arxiv.org/api/query?search_query={urllib.parse.quote_plus(arvix_query)}&amp;start=0&amp;max_results=1&#39; # Encode the search term<br>    # print(arvix_url) # Remove this to avoid printing raw url<br>    data = urllib.request.urlopen(arvix_url)<br>    # print(data.read().decode(&#39;utf-8&#39;)) # Remove this to avoid printing raw response<br>    return data.read().decode(&#39;utf-8&#39;)<br><br><br># Wikipedia search function<br>def wikipedia_search(wikipedia_query: str):<br>    &quot;&quot;&quot;Search Wikipedia.&quot;&quot;&quot;<br>    wikipedia_search_url= &#39;https://api.wikimedia.org/core/v1/wikipedia/en/search/page&#39;<br>    number_of_results = 10<br>    parameters = {&#39;q&#39;: wikipedia_query, &#39;limit&#39;: number_of_results}<br>    headers = {<br>      &#39;User-Agent&#39;: &#39;MyWikipediaApp/1.0 (icarovazquez@gmail.com)&#39;<br>    }<br><br>    wikipedia_response = requests.get(wikipedia_search_url, headers=headers, params=parameters)<br>    wikipedia_data = wikipedia_response.json() # Use wikipedia_response instead of response<br>    # pprint.pprint(wikipedia_data) # Remove this to avoid printing raw response<br>    return wikipedia_data</pre><p>With everything set up correctly, now we can start defining each member of the team.</p><h3>The Corporate Strategy Manager</h3><p>The Corporate Strategy Manager receives the research topic as a parameter, <em>research_topic</em>, and generates a step-by-step research plan and assigns an owner to each task. The code is shown below</p><pre>def corporate_strategy_manager(research_topic):<br><br>  print(&quot;==================================&quot;)<br>  print(&quot;Corporate Strategy Manager Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br>  user_prompt = f&quot;&quot;&quot;<br>    You are a Corporate Strategy manager whose job is to come up with a research plan that can be executed by other members of your team.<br>    Your team members are:<br><br>    1. A Research Assistant whose job is to find information about the research topic<br>    2. A Technical Writer whose job is summarize the research found by other agents<br>    3. An Editor whose job is to read summarize and modify them so they can distributed to the manager<br><br>    Your task is to create a step-by-step research plan as a valid Python list where each step is a string.<br><br>    Include real research-related tasks (e.g., search, summarize, draft, revise).<br>    Assume tool use is available.<br>    Do not include explanation text — return ONLY a Python list.<br>    The editor&#39;s final step should be to generate a Markdown document containing the complete research report<br><br>    Here&#39;s your research topic: &quot;{research_topic}&quot;<br>  &quot;&quot;&quot;<br><br>  messages = [{&quot;role&quot;:&quot;user&quot;, &quot;content&quot;: user_prompt}]<br><br><br>  #Call the LLM and pass the research prompt<br>  llm_response = Client.chat.completions.create(<br>      model=model,<br>      messages=messages,<br>      temperature=1,<br>      max_turns=4<br>      )<br><br><br>  steps = llm_response.choices[0].message.content.strip()<br>  #print(&quot;The plan steps inside the corporate strategy manager are: &quot;)<br>  #print(steps)<br><br>  cleaned_steps = re.sub(r&quot;^```(?:python)?\s*|\s*```$&quot;, &quot;&quot;, steps.strip(), flags=re.DOTALL)<br><br>  # Parse steps<br>  parsed_steps = ast.literal_eval(cleaned_steps)<br><br>  print(parsed_steps)<br><br>  return parsed_steps</pre><p>Let’s break down what’s going on:</p><ul><li>We create a prompt in <em>user_prompt</em> that includes the research topic entered by the user and instructions on how to go about creating the research plan. Notice we give the LLM a role, tell it what its team members are, and what the task is</li><li>Then we set up the LLM call in <em>llm_response,</em> passing the model, the prompt in the <em>messages</em> parameter, the temperature that impacts creativity, and the number of times (turns) that the LLM can use the tools</li><li>Now we call the LLM and save the response in <em>steps </em>and return <em>steps</em> for the next agent to pick up and that’s it</li></ul><p>Now we can define the next agent.</p><h3>The Research Assistant</h3><p>The Research Assistant takes each task assigned to it, executes it and returns the result. The code is shown below</p><pre>def research_agent(task: str, model: str = &quot;openai:gpt-4o&quot;, return_messages: bool = False):<br>  &quot;&quot;&quot;&quot;<br>    Execute a research task using the available tools.<br>    Returns  the assistant text, or (text, messages) if return_messages=True.<br>    &quot;&quot;&quot;<br>  print(&quot;==================================&quot;)<br>  print(&quot;🔍 Research Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br>  current_time = datetime.now().strftime(&#39;%Y-%m-%d&#39;)<br><br>  res_prompt = f&quot;&quot;&quot;<br>    You are a research assistant who can search the web, wikipedia and arxiv.<br>    You can use arxiv_tool to find academic papers, tavily_tool<br>    for general web searches and wikipedia_tool for accessing encyclopedic<br>    knowledge<br><br>    Your task is {task}<br><br>    If needed, today&#39;s date is {current_time}<br>  &quot;&quot;&quot;<br><br>  messages = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: res_prompt}]<br><br>  # Save all of your available tools in the tools list.<br>  tools = [arvix_search,<br>          tavily_search,<br>          wikipedia_search]<br><br>  # Now we call the model with the tools<br>  response = Client.chat.completions.create(<br>      model=model,<br>      messages=messages,<br>      tools = tools,<br>      tool_choice=&quot;auto&quot;,<br>      max_turns=4<br>    )<br><br>  content = response.choices[0].message.content<br>  #print(&quot;LLM response for research agent is&quot;,content)<br><br>  if return_messages:<br>    return content, messages<br>  else:<br>    return content</pre><p>Let’s go over the main parts of the code:</p><ul><li>The function receives as parameters the task created by the strategic planner manager andthe model (in this case we default to gpt 4)</li><li>We create the research prompt in <em>res_prompt </em>and add the task to the prompt</li><li>We create the <em>messages</em> param and add the <em>res_prompt</em> to it</li><li>We save all the tools available in the tools list</li><li>We create the LLM call and pass to the LLM the messages, the list of tools available to use, the tool choice set to auto which means the LLM can choose which tool to use and the number of time the LLM can use the tools</li><li>We call the LLM and catch the response in <em>content</em></li><li>We return <em>content</em> for the next agent to pick it up and use it</li></ul><p>Now we can define the next agent</p><h3>The Technical Writer</h3><p>The Technical Writer takes the summaries from the research agent and creates the first version of the report. The code is shown below.</p><pre>def technical_writer_agent(task: str):<br><br>  print(&quot;==================================&quot;)<br>  print(&quot;Technical Writer Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br>  #creates the writer prompt<br>  writer_prompt = f&quot;&quot;&quot;<br><br>  You are an expert technical writer that takes research summaries and generates<br>  technical content that can be consumed by company executives<br><br>  &quot;&quot;&quot;<br><br>  print(&quot;the task the technical writer will execute is: &quot;, task)<br><br><br>  # Define the system msg by using the system_prompt and assigning the role of system<br>  system_msg = {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: writer_prompt}<br><br>  # Define the user msg. In this case the user prompt should be the task passed to the function<br>  user_msg = {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: task}<br><br>  # Add both system and user messages to the messages list<br>  messages = [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: writer_prompt}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: task}]<br><br><br><br>  #call the LLM<br>  writer_response = Client.chat.completions.create(<br>      model=model,<br>      messages=messages,<br>      temperature=1,<br>      )<br><br><br>  pprint.pprint(writer_response.choices[0].message.content)<br><br>  print(&quot;the full dump is: &quot;)<br>  pprint.pprint(writer_response.__dict__)<br><br>  # Extract the content from the response<br><br>  content = writer_response.choices[0].message<br><br>  print(writer_response.choices[0].message)<br><br>  return content</pre><p>Let’s examine the code, it’s very similar to the previous agents’ code. What we are doing here is:</p><ul><li>We receive the task to execute as a parameter, <em>task</em></li><li>We create the instructions for the agent in <em>writer_prompt </em>and that becomes part of<em> system_msg</em></li><li>We add the task as part of the <em>user_msg</em></li><li>And we create a messages list that has both the <em>system_msg</em> and <em>user_msg</em></li><li>We then call the LLM and catch the response in <em>writer_response</em></li><li>Finally, we extract the response content from the response message and return it in <em>content</em></li></ul><p>Now we move on the define the next team member, the Editor</p><h3>The Editor</h3><p>The Editor takes the report created by the Technical Writer agent, will reflect on it, will identify areas of improvement and, finally, will write an updated version incoroporating the feedback. The code is below.</p><pre>def editor_agent(task: str):<br><br>  print(&quot;==================================&quot;)<br>  print(&quot;Editor Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br>  editor_prompt = f&quot;&quot;&quot;<br><br>    You are an expert editor agent whose task is to reflect on, critique, and/or improve drafts<br>    &quot;&quot;&quot;<br><br>  system_msg = {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: editor_prompt}<br><br>  # We define the user message which includes the user prompt. The user prompt should be the task passed to the function<br>  user_msg = {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: task}<br><br>  # Now we createa message with both prompts that we will pass to the LLM<br>  messages = [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: editor_prompt}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: task}]<br><br>  response = Client.chat.completions.create(<br>        model=model,<br>        messages=messages,<br>        temperature=1.0<br>    )<br><br>  return response.choices[0].message.content</pre><p>The code is very similar to the other agents, it has the following parts:</p><ul><li>We receive the task as the parameter <em>task</em></li><li>We write down the instructions for the editor in <em>editor_prompt </em>and make it part of<em> system_msg</em></li><li>We pass the task as part of <em>user_msg</em></li><li>We create a messages list that contains both <em>system_msg </em>and<em> user_msg</em></li><li>We call the LLM with model, messages and temperature parameters and catch the response in the <em>response</em> variable</li><li>We extract the content from the response message and return it in the <em>content</em> variable</li></ul><p>Finally, we can define the last member of our team, the Marketing Manager</p><h3>The Marketing Manager</h3><p>This agent takes the final version of the report and will create the final deliverables: a docx version of the report and slides in powerpoint format. The code is shown below. This agent is a bit more elaborate than the other agents due to the tasks it has to do.</p><pre>def marketing_agent(task:str):<br><br>  print(&quot;==================================&quot;)<br>  print(&quot;Marketing Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br><br>  output_path = &quot;research_report.docx&quot;<br>  #we convert the markdown to docx<br>  pypandoc.convert_text(<br>    md,<br>    to=&quot;docx&quot;,<br>    format=&quot;md&quot;,<br>    outputfile=output_path<br>    )<br><br>  #now we generate slides using slidesgpt&#39;s api<br>  headers = {<br>        &quot;Authorization&quot;: f&quot;Bearer {os.environ[&#39;SLIDESGPT_API_KEY&#39;]}&quot;,<br>        &quot;Content-Type&quot;: &quot;application/json&quot;,<br>    }<br><br>  #getting rid of the markdown<br>  plain_text = markdownify(md)<br><br>  #we have to trim our text because it&#39;s too long for the slidegpt api<br>  # trim to &lt; 3500 tokens<br>  enc = tiktoken.get_encoding(&quot;cl100k_base&quot;)<br>  tokens = enc.encode(plain_text)<br><br>  MAX_TOKENS = 3000<br>  if len(tokens) &gt; MAX_TOKENS:<br>    tokens = tokens[:MAX_TOKENS]<br>    plain_text = enc.decode(tokens)<br><br>  print(&quot;Length:&quot;, len(tokens), &quot;tokens&quot;)<br><br>  slides_prompt = f&quot;&quot;&quot;<br>  Create a professional PowerPoint presentation based on the following research content.<br><br>  Length:<br>  - 12–18 slides<br>  - Use headings, bullet points, and concise explanations<br><br>  Audience:<br>  - Executives<br><br>  Content: {plain_text}<br><br>  &quot;&quot;&quot;[:8000]<br><br>  resp = requests.post(<br>      slidesgpt_api_endpoint,<br>      headers=headers, json={&quot;prompt&quot;:slides_prompt})<br>  print(&quot;Status code:&quot;, resp.status_code)<br>  print(&quot;Response text:&quot;, resp.text)  # IMPORTANT<br>  resp.raise_for_status()<br>  result = resp.json()<br>  download_url = result[&quot;download&quot;]<br>  pptx_resp = requests.get(download_url)<br>  pptx_resp.raise_for_status()<br><br>  return pptx_resp.content</pre><p>What’s going on in the code? Let’s take a look:</p><ul><li>As with all the other agents, we receive the task in the <em>task</em> parameter</li><li>We then take the report created by the editor, which is in md (we will explain this later), and convert it to docx using the <em>pypandoc</em> library. The new file is in name defined in <em>output_path</em></li><li>Now we can create the slides by using the slidesgpt API</li><li>For that we need to first setup authentication using a Bearer token we got from the slidesgpt API console. We set that in the <em>headers</em> var</li><li>Then we have to trim our prompt because slidesgpt only accepts a maximum of 3,000 tokens</li><li>Once we do that we can create our prompt. The instructions for the marketing manager are in the slides_prompt where we mention we want a presentation of 12–18 slides that summarizes the research contenct we provide in the <em>tokens</em> var</li><li>Now we can call the slidesgpt API using requests.post and pass on as parameters the API endpoint (defined earlier in the code), the headers and the prompt in JSON format</li><li>We catch the API response in the <em>resp</em> parameter</li><li>We then extract the download URL from the response and do an http get to download the slides. We store it in the <em>pptx_resp.content</em> var and return it and that’s it.</li></ul><p>Now that we defined all members of the team that will execute tasks we can finally move on to the last member of the team: a Project Manager that will make sure all the tasks are executed.</p><h3>The Project Manager</h3><p>Before defining the Project Manager, we need to list all the agents the Project Manager will track. We do that in the code below.</p><pre>list_of_agents = {<br>    &quot;research_agent&quot;: research_agent,<br>    &quot;editor_agent&quot;: editor_agent,<br>    &quot;technical_writer_agent&quot;: technical_writer_agent,<br>    #&quot;marketing_agent&quot;: marketing_agent<br>}</pre><p>Now we can define the Project Manager, its code is shown below. As you can see, it’s a bit more elaborate than the other agents’ code.</p><pre>def project_manager_agent(topic, model: str = &quot;openai:gpt-4o&quot;, limit_steps: bool = True):<br><br>  print(&quot;==================================&quot;)<br>  print(&quot;Project Manager Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br>  plan_steps = corporate_strategy_manager(topic)<br>  print(&quot;The plan steps from the corporate strategy manager are: &quot;)<br>  pprint.pprint(plan_steps)<br>  max_steps = 15<br><br>  #this will hold a list of the tasks and agents that have already been executed<br>  history=[]<br><br>  for i, step in enumerate(plan_steps):<br><br>    agent_assignment_prompt = f&quot;&quot;&quot;<br>        You are a project manager and an execution manager for a multi-agent research team.<br><br>        Given the instruction below, identify which agent should perform it and extract the clean task.<br><br>        Return only a valid JSON object with two keys:<br>        - &quot;agent&quot;: one of [&quot;research_agent&quot;, &quot;editor_agent&quot;, &quot;technical_writer_agent&quot;]<br>        - &quot;task&quot;: a string with the instruction that the agent should follow<br><br>        Only respond with a valid JSON object. Do not include explanations or markdown formatting.<br><br>        Instruction: &quot;{step}&quot;<br><br>        &quot;&quot;&quot;<br>    response = Client.chat.completions.create(<br>        model=model,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: agent_assignment_prompt}],<br>        temperature=0,<br>        )<br><br><br>    raw_content = response.choices[0].message.content.strip()<br><br>    #cleaning up any markdown<br>    clean_json = re.sub(r&quot;^```(?:json)?\s*|\s*```$&quot;, &quot;&quot;, raw_content.strip())<br><br>    try:<br>      agent_content = json.loads(clean_json)<br>    except json.JSONDecodeError:<br>      print(&quot;⚠️ Could not decode JSON. Raw output:&quot;)<br>      print(raw_content)<br><br><br>    print(&quot;The formatted agent_content is &quot;)<br>    pprint.pprint(agent_content)<br><br>    agent_name = agent_content[&quot;agent&quot;]<br>    task_name = agent_content[&quot;task&quot;]<br><br>    context = &quot;\n&quot;.join([<br>            f&quot;Step {j+1} executed by {a}:\n{r}&quot;<br>            for j, (s, a, r) in enumerate(history)<br>        ])<br><br>      #Here is the context of what has been done so far:<br>      #  {context}<br><br>    enriched_task = f&quot;&quot;&quot;<br>      You are {agent_name}.<br><br>      Here is the context of what has been done so far:<br>        {context}<br><br>      Your next task is:<br>        {task_name}<br><br>      &quot;&quot;&quot;<br><br>    print(f&quot;\n Agent: `{agent_name}` executing task: {enriched_task}&quot;)<br><br>    if agent_name in list_of_agents:<br>      output = list_of_agents[agent_name](enriched_task)<br>      history.append((step, agent_name, output))<br>    else:<br>      output = f&quot; Unknown agent: {agent_name}&quot;<br>      text_output = output.content if hasattr(output, &quot;content&quot;) else str(output)<br>      history.append((step, agent_name, output))<br><br>    print(f&quot;✅ Output:\n{output}&quot;)<br><br>  return history</pre><p>Let’s go over what’s going on here.</p><ul><li>The Project Manager agent receives the research topic as the <em>topic</em> parameter</li><li>We define the maximum number of steps the research plan can have. I set a limit of 15 basically so we don’t run into the tokens/minute limit most LLMs have since we will be executing the whole pipeline at once and we will be calling the same LLM a few times</li><li>We also initialize a history array that will keep a record of everything that was executed</li><li>Then we do a for loop where we execute one task every turn</li><li>In the loop we create the <em>agent_assignment_prompt </em>assigning the LLM the role of project manager and telling it to return, in JSON format, the agent and the task the agent will execute. It also includes the step that has the task to execute</li><li>Then we call the LLM with the <em>agent_assignment_prompt </em>and catch the response in the <em>response</em> variable</li><li>We then clean the JSON to make sure it’s valid JSON</li><li>We assign the agent and the task to the <em>agent_name</em> &amp; <em>task_name</em> variables</li><li>Then we create a prompt, <em>enriched_task</em>, with the <em>agent_name</em> &amp; <em>task_name</em></li><li>Then we invoke the appropriate agent by using the list of agents array we created earlier and catch the <em>output</em> in the output variable</li><li>Finally, we add the output to the history and we print it so we can see, step by step, how the plan is progressing</li></ul><p>Now we can run the whole plan</p><h3>Running the Whole Research Plan</h3><p>To execute the Research Plan we have to have a research topic. So we start by asking the end user for one. The code below shows that a few more things.</p><pre>research_topic = input(&quot;Please enter the topic you want this team to research:&quot;)<br>print(f&quot;You entered the following: {research_topic}&quot;)<br><br><br>executor_history = project_manager_agent(research_topic, limit_steps=True)<br><br>md = executor_history[-1][-1].strip(&quot;`&quot;)<br>display(Markdown(md))<br><br>#call the marketing agent to generate the powerpoint slides<br>slides_content = marketing_agent(md)<br>with open(&quot;research_report.pptx&quot;, &quot;wb&quot;) as f:<br>    f.write(slides_content)</pre><p>Let’s go through it:</p><ul><li>First we ask the user to enter a research topi using <em>input</em></li><li>Then we kick off our agentic pipeline by invoking the project manager agent and passing as a parameter the <em>research_topic</em> entered by the user and we catch the output in <em>executor_history</em></li><li>Then we get trid of some spaces on the output, store it in the <em>md</em> variable and display it</li><li>Finally we call the marketing agent to so it can generate the docx and ppt files.</li></ul><p>Let’s look at the output in detail. So we first ask the end user what is the research topic and in Figure 1 below we entered one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hSd7UmtdnOVjTM-XjPb9ZQ.png" /><figcaption><strong>Figure 1. Kicking off the Agentic Pipeline by entering a Research Topic</strong></figcaption></figure><p>Notice our topic is if Netflix should buy Warner Discovery. Warner Discovery put itself for sale after multiple offers from Paramount SkyDance and Netflix was one of the bidders. Although their bid was accepted by Warner Discovery, Paramount launched a hostile bid so the story is not over yet. So it’s kind of relevant we are asking this question. Imagine you are part of the Netflix team deciding if you should bid for Warner Discovery or you are an external market analyst or say, a Business School student curious about the transaction. In all cases, one of the first questions you ask is if Netflix should buy Warner and for that you need to do some research. A perfect topic for our team of AI agents to tackle.</p><p>Let’s see what kind of research plan our Corporate Strategy Mananger came up with. The plan is shown in the next 2 images (couldn’t fit in one image :-( )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QDUGo0Hg2J0PxY-Kg3IGpQ.png" /><figcaption><strong>Figure 2. First part of the Research Plan created by our Corporate Strategy Manager Agent</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yYAUzwkn5p3aqolv-OjMNQ.png" /><figcaption><strong>Figure 3. Second part of the Research Plan created by our Corporate Strategy Manager Agent</strong></figcaption></figure><p>Notice the number of steps in the Research Plan is 15, which is the limit we specified. We could have gone higher but we were already bumping into the max tokens/minute limit that Cluade has. The plans looks pretty good. Now let’s take a look at the some of the output of some of the steps. It would take too many screenshots to show the whole thing so if you want to see the whole thing it’s better that you download the code from the <a href="https://github.com/icarovazquez/agentic-ai/blob/main/Generic_Corporate_Strategy_Team.ipynb">repo</a> and try it on your own.</p><p>Here’s a couple of screenshots:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O77y_eWxqxb4EgI75zdUPA.png" /><figcaption><strong>Figure 4. Research Agent executing a task and partial output of that task</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RW4BYXezUrtoViGC1r49eg.png" /><figcaption><strong>Figure 5. Editor Agent Idenitfying areas of improvement for the first version of the research report</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tjg9-YruqjjL-Ssypyv5Ng.png" /><figcaption><strong>Figure 6. A chunk of the final Research Report rewritten by the Editor agent</strong></figcaption></figure><p>Next, the marketing manager agent creates the two files containg the report and the slides. We can download them with the code below.</p><pre>files.download(&quot;research_report.docx&quot;)<br>files.download(&quot;research_report.pptx&quot;)</pre><p>I decided on a docx format because it can be opened on Word, Google Docs or Pages, Similarly, I picked a pptx format for the slides because it can be opened in Powerpoint, Google Slides or Keynote. Figures 7and 8below show a couple of screenshots of the two files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B37BxIFzfu8hPw_VvXF7pg.png" /><figcaption><strong>Figure 7. A partial view of the final Research report in Word</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zkte4KTyziE74R7k91a-sQ.png" /><figcaption><strong>Figure . A slide from the presentation created by the Marketing Manager</strong></figcaption></figure><p>I uploaded both files to Google Drive. If you want to see the content on both files, you can do so here:</p><ul><li>Here’s the <a href="https://docs.google.com/presentation/d/1OqKqkt9gE3prx52p1pibfY8dgstus9LZ/edit?usp=sharing&amp;ouid=114208909225278773092&amp;rtpof=true&amp;sd=true">link for the powerpoint slides</a> and</li><li>Here’s the <a href="https://docs.google.com/document/d/1XcUwTGFpFwuoWA9Bf_820SnfsH37PWZi/edit?usp=drive_link&amp;ouid=114208909225278773092&amp;rtpof=true&amp;sd=true">link for the word-formatted research report</a></li></ul><p>So what should we think of the output? If you do a quick glance or read them in their entirety I think you’ll agree it’s a pretty good effort for a 1st version of both files. This is how I think of anything AI produces these days: It’s a first try and then you, the human, revises the output and refines it. In this case I would make extensive changes (change the tone of the document, add graphics/charts to the slides, reduce the text on slides etc.) to both but the skeleton is there.</p><h3>Possible Improvements</h3><p>Nothing is perfect and this Agentic AI code can be improved in many ways. Some of them are:</p><ul><li><em>Evals</em>: I haven’t added any evaluations for each of the steps or agents. The fact that the research plan can change makes this harder but we would just run the same topic multiple times and measure the quality of the output for each time.</li><li><em>Better Marketing Manager Integration</em>: Right now the marketing manager is kind of outside the main pipeline execution because it’s out of the loop. It was not added to the list of agents the Corportate Strategy Manager and the Project Manager can assign tasks to. This is for a couple of reasons: the Marketing Manager has a very well defined task (create 2 files) and adding it to the main pipeline would mean the tasks assigned to it could be anything. Furthermore, assigning those kinds of tasks to an LLM is not a good thing since most LLMs today cannot generate slides. They can generate slides-style content but not powerpoint files, for example.</li><li><em>Cleanup</em>: We have a bunch of debugging statements like prints that can be removed</li><li><em>Production Deployment</em>: We can run the code in a notebook but to deploy it to a production environment we need a different artifact, like an API. We can do that in a future post.</li></ul><h3>Conclusion</h3><p>Having a team of agents at your beck and call is a great way to increase our productivity on a day-to-day basis. You must have heard everywhere that AI is going to replace us and will do everything. Maybe but not anytime soon :-) Right now it’s a companion to help us do stuff faster.</p><p>As you can see from the code, it’s not hard to create your own Agentic AI team. You should do it so you can be more productive and be on par with all the Silicon Valley grandmothers and Uber drivers that have already created their own Agentic AI teams :D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9140b73027d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI: A Quick Overview & An Example]]></title>
            <link>https://medium.com/@icaro_vazquez/agentic-ai-a-quick-overview-an-example-3aa538926e53?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3aa538926e53</guid>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Tue, 28 Oct 2025 18:45:26 GMT</pubDate>
            <atom:updated>2025-12-02T22:34:02.542Z</atom:updated>
            <content:encoded><![CDATA[<p>Unless you have been under a rock for the last year, you must have heard the term Agentic AI, and if you are here in Silicon Valley, you really hear it all the time, from the Lyft Driver to the gardener to all the tech people you work with; heck, most likely all local grandmothers are building their own agents right now. And, at some point, you asked yourself: What the f are they talking about?</p><p>I will try to explain what it is in plain English and provide an example of its capabilities. The first part will keep the tech jargon to a minimum, but the second part will be full of it since it’s going to be mostly code :-)</p><h3><strong>What is it?</strong></h3><p>This will be a very basic overview; there are many sources that can give you a more comprehensive explanation. I’ll just cover the main points. You most likely heard of Generative AI (or just AI), and you are using it any time you use something like ChatGPT. When you use ChatGPT, it’s like you are having a conversation with an assistant: you ask a question, get a response, then ask a follow-up question, and so on.</p><p>I’m going to say Agentic AI is basically an extension of you using ChatGPT. Instead of using a chat interface, you are using the Chatbot backend, a Large Language Model (LLM), through code. I’m going to give you my definition and then the definition of the guy who invented the term.</p><p>My definition:</p><blockquote>Agentic AI is using LLMs and some tools (most of the time, APIs) to complete an ask or a task.</blockquote><p>Now, the guy who invented the term, Andrew Ng, defines it as:</p><p><em>A process where an LLM-based workflow or pipeline executes multiple steps to complete a task</em></p><p>So it’s basically asking the LLM (or various LLMs) and associated tools to complete a task for you. I think the best way to explain it is with an example. Let’s go through an example, and we will mention, at the end, why Agentic AI can help you be more productive.</p><h3><strong>An Example: A Corporate Research Team</strong></h3><p>Imagine you work in a Corporate Strategy group at a big company in 2025, and you are tasked with writing a report for your manager about the impact of the tariffs the Trump Administration is implementing: what will be their impact on the economy? What has happened before when someone tried to do the same thing? What should we do about it, if anything?</p><p>Before GenAI &amp; Agentic AI, you would have to do things the slow way:</p><ul><li>You would do a bunch of web searches, look at economics academic papers, Wikipedia, and so on, or ask someone else to do all the research for you.</li><li>You, or your researcher, then create a summary of your research findings and pass it on to someone else, perhaps you, to write the first version of the report</li><li>That first version of the report will be edited by someone, either you or another member of the team</li><li>Someone will add some graphics and charts to make the report more digestible.</li></ul><p>This process will take days, if not weeks, because you are busy with other asks, the other team members are also busy with other projects, and so on. So what can you do to complete this project faster? You use Agentic AI to create your own team of agents that will do everything (well, not everything, but a lot!) for you in literally seconds. Let me show how.</p><h4>The Code</h4><p>I created this Agentic AI example in a Google Colab Notebook, so it might have some hooks that are Colab-specific, but most of the code can be taken and run in any notebook instance.</p><p>So what are we going to do? We are going to create a team of 4 agents that will do the following:</p><ul><li>A Research Assistant whose job is to find information about the US economy and tariffs</li><li>A Technical Writer whose job is to write papers</li><li>A Graphical Designer whose job is to create images</li><li>A Marketing Manager whose job is to collate outputs from the previous members and produce the final version of the report</li></ul><p>The Agentic AI team will use the following external tools:</p><ul><li>Tavily: to do web searches</li><li>Arxiv: to search academic papers</li><li>Wikipedia: for relevant article searches</li></ul><p>To create the team, we will use various Python libraries:</p><ul><li><a href="https://github.com/andrewyng/aisuite">aisuite</a>: A library created by Andrew Ng that allows us to easily interact with various LLM providers</li><li>openai: to call Dall-e, which will generate the images for the report</li><li>anthropic: We are going to use Claude as the main LLM</li></ul><p>The LLMs and tools need a valid API key since we are going to use those APIs. This means you have to go to each provider, create an account, create an api key, and import that API key into the notebook. There are many ways of doing this, but the one we used here is specific to Google Colab.</p><h4>Tool Setup</h4><p>There is a secrets section of the Notebook that can be accessed by clicking on the key icon that is in the left navigation menu, as shown in Figure 1 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/104/1*2hBStzn8eXiPcGaWdmZ82Q.png" /><figcaption><strong>Figure 1. Secrets Icon In Google Colab</strong></figcaption></figure><p>Clicking on the icon opens a panel where I can add all the API keys I need and decide which ones the notebook will have access to. Figure 2 below shows my list of API Keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*aatR6YdKu68c-LyEUlqRSQ.png" /><figcaption><strong>Figure 2. Secrets Panel with a list of available secrets</strong></figcaption></figure><p>As you can see, I have 8 API Keys, and the notebook has access to 5 of them. They are:</p><ul><li>Anthropic: So we can call Claude using aisuite</li><li>Google AI Studio: So we can use Gemini</li><li>OpenAI: So we can use OpenAI models like GPT or DALL-E</li><li>Tavily: So we can enable our AI agents to do web searches</li><li>Wikipedia: I created a Wikipedia API key, but you don’t need it to search; you only need it to write or update articles</li></ul><p>The goal is to have the research report (or a decent version of it) ready at the end of this process. In this case, we will generate a PDF version so it can be easily seen in the browser.</p><p>Let’s go step by step:</p><p>First, we start by installing all the libraries needed:</p><pre>#install needed libs<br>!pip install aisuite<br>!pip install anthropic<br>!pip install tavily-python<br>!pip install openai<br>!pip install reportlab<br>!pip install fpdf</pre><p>Now we import all that we need into our environment:</p><pre>#import all the needed libs<br>import base64<br>import json<br>import os<br>import re<br>from datetime import datetime<br>from google.colab import userdata<br>from io import BytesIO<br>import pprint<br>import requests<br>import textwrap<br><br># 3rd Party<br>from PIL import Image<br>from IPython.display import display, Image as ImageDisplay, IFrame<br>import aisuite as ai<br>import urllib, urllib.request<br>import urllib.parse # Import urllib.parse<br>import openai<br>from tavily import TavilyClient<br>from reportlab.lib import colors<br>from reportlab.lib.pagesizes import letter<br>from reportlab.pdfgen import canvas<br>from reportlab.lib.utils import ImageReader<br>from google.colab import files<br>from fpdf import FPDF</pre><p>Once we do that, we can import the API Keys we are going to use as follows:</p><pre>os.environ[&#39;ANTHROPIC_API_KEY&#39;] = userdata.get(&#39;ANTHROPIC_API_KEY&#39;)<br>os.environ[&#39;TAVILY_API_KEY&#39;] = userdata.get(&#39;TAVILY_API_KEY&#39;)<br>os.environ[&#39;OPENAI_API_KEY&#39;] = userdata.get(&#39;OPENAI_API_KEY&#39;).strip()</pre><p>Now the API Keys are available for aisuite to use. We can initialize all the clients we need and set the URLs we are going to call. Below, we initialize:</p><ul><li>The aisuite client we are going to use to call the LLMs</li><li>The tool clients (tavily, arvix &amp; wikipedia) and</li><li>We set the LLM to claude 3.7 sonnet</li></ul><pre>#initialize ai client<br>Client = ai.Client()<br><br>#Initializing research tools (tavily, arvix &amp; wikipedia)<br><br>#initialize tavily client<br>tavily_client = TavilyClient(api_key=os.environ[&#39;TAVILY_API_KEY&#39;])<br><br><br>#set arvix sample search term<br>arvix_search_term = &#39;impact of tariffs in the economy&#39;<br>arvix_url = f&#39;http://export.arxiv.org/api/query?search_query={urllib.parse.quote_plus(arvix_search_term)}&amp;start=0&amp;max_results=1&#39; # Encode the search term<br><br><br>#set wikipedia url<br>wikipedia_search_url= &#39;https://api.wikimedia.org/core/v1/wikipedia/en/search/page&#39;<br>headers = {<br>  &#39;User-Agent&#39;: &#39;MyWikipediaApp/1.0 (icarovazquez@gmail.com)&#39;<br>}<br><br># We are going to use claude for this team<br>model=&quot;anthropic:claude-3-7-sonnet-20250219&quot;</pre><p>Now, just to be sure, we are going to do a sample API call to each tool to make sure our API calls are working, and we can get results from each tool. First, Tavily:</p><pre>#We ask Tavily to ask a simple question<br>search_response = tavily_client.search(&quot;Who is the chairman of the Federal Reserve Bank?&quot;)<br>pprint.pprint(search_response)</pre><p>As you can see, as a test, we are asking Tavily to do a web search for who is the chairman of the Federal Reserve Board. I won&#39;t show all the results we receive, but here’s a sample of what we get back. You may see results similar to those in Figure 3 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0RK86escrBGP-9ZoLiN-iA.png" /><figcaption><strong>Figure 3. Partial resuts returned by making a Tavily API call</strong></figcaption></figure><p>OK, Tavily is working. Now we do the same for <a href="https://arxiv.org/">arXiv</a>, a repository of research papers for various topics, hosted by Cornell University, including economic papers. Here’s the call to arvix:</p><pre>print(arvix_url)<br>data = urllib.request.urlopen(arvix_url)<br>print(data.read().decode(&#39;utf-8&#39;))</pre><p>We print the arXiv URL just to see what we are calling, and a partial view of the results is shown in Figure 4 below. Notice our arXiv query is about the impact of tariffs on the economy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ff7dhQOx-2DCSNRMqle9Qw.png" /><figcaption><strong>Figure 4. Partial results by making an arXiv API call</strong></figcaption></figure><p>Cool, arXiv is working. Now we just do a sample API call to Wikipedia to make sure it’s working:</p><pre>wikipedia_search_query = &#39;economic tariffs&#39;<br>number_of_results = 10<br>parameters = {&#39;q&#39;: wikipedia_search_query, &#39;limit&#39;: number_of_results}<br><br><br>wikipedia_response = requests.get(wikipedia_search_url, headers=headers, params=parameters)<br>wikipedia_data = wikipedia_response.json() # Use wikipedia_response instead of response<br>pprint.pprint(wikipedia_data)</pre><p>Executing the notebook cell will get your results like the ones shown in Figure 5 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J_KVpbDccwEEIt7iermxDA.png" /><figcaption><strong>Figure 5. Partial results by making a Wikipedia API call</strong></figcaption></figure><p>Cool, all 3 tool API calls are working. Now we move on to create functions for each one. Why functions? Because we will make the tools available to the LLM, and we need to pass them as functions. Below is the code for the function definitions for all 3 tools.</p><pre># Tavily web search function<br>def tavily_search(web_query: str):<br>    &quot;&quot;&quot;Search the web using Tavily.&quot;&quot;&quot;<br>    search_response = tavily_client.search(web_query)<br>    # pprint.pprint(search_response) # Remove this to avoid printing raw response<br>    # data = search_response.json() # Tavily response is already a dictionary<br>    return search_response<br><br># Arvix search function<br>def arvix_search(arvix_query: str):<br>    &quot;&quot;&quot;Search academic papers with Arvix &quot;&quot;&quot;<br>    arvix_url = f&#39;http://export.arxiv.org/api/query?search_query={urllib.parse.quote_plus(arvix_query)}&amp;start=0&amp;max_results=1&#39; # Encode the search term<br>    # print(arvix_url) # Remove this to avoid printing raw url<br>    data = urllib.request.urlopen(arvix_url)<br>    # print(data.read().decode(&#39;utf-8&#39;)) # Remove this to avoid printing raw response<br>    return data.read().decode(&#39;utf-8&#39;)<br><br><br># Wikipedia search function<br>def wikipedia_search(wikipedia_query: str):<br>    &quot;&quot;&quot;Search Wikipedia.&quot;&quot;&quot;<br>    wikipedia_search_url= &#39;https://api.wikimedia.org/core/v1/wikipedia/en/search/page&#39;<br>    number_of_results = 10<br>    parameters = {&#39;q&#39;: wikipedia_query, &#39;limit&#39;: number_of_results}<br>    headers = {<br>      &#39;User-Agent&#39;: &#39;MyWikipediaApp/1.0 (icarovazquez@gmail.com)&#39;<br>    }<br><br>    wikipedia_response = requests.get(wikipedia_search_url, headers=headers, params=parameters)<br>    wikipedia_data = wikipedia_response.json() # Use wikipedia_response instead of response<br>    # pprint.pprint(wikipedia_data) # Remove this to avoid printing raw response<br>    return wikipedia_data</pre><p>Now we can move on to define our team of agents</p><h3>Agent Team Definition</h3><h4>Research Manager Agent</h4><p>We will start by defining the Research Manager Agent. This agent’s task is to do research using the available tools and write an initial version of the report. Here’s the agent definition code:</p><pre>def research_assistant_agent(tools):<br><br>  print(&quot;==================================&quot;)<br>  print(&quot;Research Assistant Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br>  res_prompt = f&quot;&quot;&quot;<br>    You are an expert economics research agent and you need to prepare an analysis of the effect of tariffs in the US economy.<br><br>    Your goals is:<br>    1. Explore current economics trends related to tariffs using web search<br>    2. Summarize your findings<br>    3. Provide a report that includes your findings<br>    4. If needed, today date is {datetime.now().strftime(&quot;%Y-%m-%d&quot;)}.<br><br>    You can call the following tools:<br>    1. tavily_search to discover external web trends<br><br>    Once you complete your analysis, write a report that mentions:<br>    * The top 3 findings you uncovered<br>    * Past scenarios wheere tariffs were enforced and what was the result for the economy<br>    * Your assessment of what tariffs will do to the US economy and the US stock market if enforced<br>    * A list of references you used<br>  &quot;&quot;&quot;<br><br>  messages = [{&quot;role&quot;:&quot;user&quot;, &quot;content&quot;: res_prompt}]<br><br><br><br>  #Call the LLM and let it know what tools are available<br>  llm_response = Client.chat.completions.create(<br>      model=model,<br>      messages=messages,<br>      tools=tools, # Pass the formatted tool definitions here<br>      temperature=1,<br>      #tool_choice=&quot;auto&quot;,<br>      max_turns=6,<br>      verbose=True<br>      )<br><br>  pprint.pprint(llm_response.choices[0].message.content)<br><br>  print(&quot;the full dump is: &quot;)<br>  pprint.pprint(llm_response.__dict__)<br><br>  # Extract the content from the response<br><br>  content = llm_response.choices[0].message<br><br>  return content</pre><p>Many things are happening here, so let’s list them:</p><ul><li>We define the prompt we are giving to the LLM in r<em>es_prompt</em>. We are assigning an expert role to the LLM and asking it to search the web, find information about the effects of tariffs on the economy, and summarize the findings in a report</li><li>Then we use the aisuite client we initialized earlier to make a call to the LLM. Notice the parameters we use: <em>model</em> is the LLM we are going to use (Claude in this case, we defined it earlier in the code), <em>messages</em> is the prompt we just defined, <em>tools</em> is the list of tools we want the LLM to use (this list is a function parameter that will be passed once the function is called), <em>temperature</em> (range from 0.0 to 2.0), specifies how creative we want the LLM to be in terms of the output it produces; lower means less creative, higher means more creative, <em>max_turns</em> (tells the LLM how many times it can use the available tools) &amp; verbose (to get more detailed logs/output)</li><li>Finally, we make the call and assign the response to the <em>content</em> variable and return that variable as the result of the function</li></ul><p>We are now ready to call our newly defined Research Agent. We invoke it using the following code:</p><pre># Create a dictionary of tool functions<br>tool_functions = [<br>    tavily_search,<br>    arvix_search<br>]<br><br># Pass the tool functions to the agent<br>research_assistant_result= research_assistant_agent(tool_functions)<br>print(&quot;The result of the assistant is &quot;)<br>pprint.pprint(research_assistant_result)</pre><p>Simple steps here:</p><ul><li>We first define the list of available tools for the LLM. In this case, we are saying Tavily and Arvix are available for the Research Agent to use in its research.</li><li>Then we call the <em>research_assistant_agent</em> function we defined in the previous step with the tool list we just created and store the result in <em>research_assistant_result</em>.</li><li>Finally, we print the results to see what the LLM came up with. These results are shown in Figure 6 below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*joT_inkB5pbXDUpXIKU1zQ.png" /><figcaption><strong>Figure 6. Partial Research Agent Results</strong></figcaption></figure><p>I didn’t paste the whole essay, but we can see the Research Assistant wrote a first version of the report, great! Now we can move on to define the 2nd agent: the Editor Agent, who will take the output from the Research Assistant, reflect on it, suggest improvements, and write an updated version of the report.</p><h4>Writer Agent</h4><p>Let’s start with the Writer Agent definition, shown below</p><pre>def writer_agent(research_insights: str):<br><br>  print(&quot;==================================&quot;)<br>  print(&quot;Writer Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br>  #creates the writer prompt<br>  writer_prompt = f&quot;&quot;&quot;<br><br>  You are an experienced writer that takes writing drafts, reflects on them and expands and<br>  summarizes them.<br><br>  &quot;&quot;&quot;<br><br>  user_prompt= f&quot;&quot;&quot;<br><br>  the research insights are: {research_insights}<br><br>  Please reflect on these insights, tell us what needs to change and why and<br>  modify the report draft as you think is neeeded<br><br>  Please output a modified version of the report draft and<br>  return your response in JSON with the following keys:<br><br>  {{<br>    &quot;reflection&quot;: &quot;...&quot;,<br>    &quot;rewritten_essay&quot;: &quot;...&quot;<br>  }}<br><br>  Return **only valid JSON**, properly escaped, with no extra text, Markdown, or commentary.<br><br>  &quot;&quot;&quot;<br><br>  messages = [{&quot;role&quot;:&quot;system&quot;, &quot;content&quot;: writer_prompt},<br>              {&quot;role&quot;:&quot;user&quot;, &quot;content&quot;: user_prompt}]<br><br><br>  #model = &quot;openai:gpt-3.5-turbo&quot;<br><br><br>  #call the LLM<br>  writer_response = Client.chat.completions.create(<br>      model=model,<br>      messages=messages,<br>      temperature=1,<br>      )<br><br>  pprint.pprint(writer_response.choices[0].message.content)<br><br>  print(&quot;the full dump is: &quot;)<br>  pprint.pprint(writer_response.__dict__)<br><br>  # Extract the content from the response<br><br>  content = writer_response.choices[0].message<br><br>  print(writer_response.choices[0].message)<br><br>  return content</pre><p>Let’s go over the major parts of this agent:</p><ul><li>System Prompt: We give the writer agent a role</li><li>User Prompt: Here, we include the Research Agent output as <em>research_insights</em> and ask the Writer Agent to reflect on the findings, suggest improvements, and rewrite the report</li><li>We also asked the agent to return its write-up in properly escaped JSON format</li><li>We now create the messages payload with the 2 prompts: the system prompt, <em>writer_prompt</em>, and the user prompt, <em>user_prompt</em></li><li>We call the LLM with the model (still Claude), messages (the messages payload we just defined), and temperature (set to 1) parameters, and get its output in the <em>writer_response</em> var</li><li>Finally, we get the part of the output we are interested in to the <em>content</em> var and return the <em>content</em> var.</li></ul><p>Now we can call the Writer Agent and see what results it provides. We call it with the following code</p><pre>writeragent_result = writer_agent(<br>    research_insights=research_assistant_result,<br>)<br><br>print(&quot;the content string is &quot;)<br>print(writeragent_result.content)<br><br>response_text = str(writeragent_result.content)<br><br>#trying to extract JSON from the object<br>json_match = re.search(r&#39;\{.*\}&#39;, response_text, re.DOTALL)<br>if not json_match:<br>    raise ValueError(&quot;No JSON object found in the response:\n&quot; + response_text)<br><br>json_text = json_match.group(0)<br>json_text = json_text.strip()<br><br><br># Clean invalid control characters (ASCII &lt; 32)<br>json_text = re.sub(r&#39;[\x00-\x1f]+&#39;, &#39; &#39;, json_text)<br><br># Remove any trailing commas before a closing brace or bracket<br>json_text = re.sub(r&quot;,(\s*[}\]])&quot;, r&quot;\1&quot;, json_text)<br><br># Escape unescaped quotes inside string values<br>json_text = json_text.replace(&#39;\n&#39;, &#39;\\n&#39;)<br><br><br>try:<br>  data = json.loads(json_text)<br>except json.JSONDecodeError as e:<br>    print(&quot;JSON parsing failed:&quot;, e)<br>    print(&quot;Cleaned JSON text was:\n&quot;, json_text)<br>    raise<br><br>reflection_part = data[&quot;reflection&quot;]<br>essay_part = data[&quot;rewritten_essay&quot;]<br><br>print(&quot;Reflection:\n&quot;, reflection_part)<br>print(&quot;Essay:\n&quot;, essay_part)</pre><p>The major parts are:</p><ul><li>Invoke the agent and store its results in the <em>writeragent_result</em> var</li><li>Convert the response to a string stored in <em>response_text</em></li><li>Do some JSON formatting and store the results in the <em>reflection_part</em> &amp; <em>essay_part</em> vars</li></ul><p>And the result is as shown in Figures 7 and 8 below. First, a snippet of the reflection part:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XOkF7e7ygnBLbP6L4c_1Lw.png" /><figcaption><strong>Figure 7 Partial Reflection returned by the Writer Agent</strong></figcaption></figure><p>and a snippet of the rewritten report:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1wwvYLUYCiYREoRk6sDT5A.png" /><figcaption><strong>Figure 8. Partial Rewritten essay returned by the Writer Agent</strong></figcaption></figure><p>Excellent! Now we can move on to generating an image that will make our report more attractive. For that, we will create a graphical designer agent. This agent takes as input the report and uses it to generate an image relevant to the report subject.</p><h4>Graphical Designer Agent</h4><p>We are asking a Graphical Designer Agent to create an image, but Claude, the LLM we have been using until now, does not generate images from text, so we need to change LLMs. We are going to use OpenAI’s DALL-E, which supports this capability. We define our agent in the code below.</p><pre>def designer_agent(essay_summary: str, size: str = &quot;1024x1024&quot;):<br><br>  print(&quot;==================================&quot;)<br>  print(&quot;Graphical Designer Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br>  #creates the designer prompt<br>  designer_prompt = f&quot;&quot;&quot;<br><br>  You are an experienced graphical designer that takes essays and generate graphics and charts based on the text.<br>  Based on this essay: {essay_summary}<br><br>  Please create an image that does not include text and is appropriate for this subject<br>  &quot;&quot;&quot;<br><br><br>  openai_client = openai.OpenAI(<br>    api_key=os.environ[&#39;OPENAI_API_KEY&#39;].strip())<br><br>  image_response = openai_client.images.generate(<br>        model=&quot;dall-e-3&quot;,<br>        prompt=designer_prompt,<br>        size=size,<br>        quality=&quot;standard&quot;,<br>        n=1,<br>        response_format=&quot;url&quot;<br>    )<br><br>  image_url = image_response.data[0].url<br>  print(image_url)<br><br>  # Save image locally<br>  img_bytes = requests.get(image_url).content<br>  img = Image.open(BytesIO(img_bytes))<br><br>  filename = os.path.basename(image_url.split(&quot;?&quot;)[0])<br>  print(filename)<br>  image_path = filename<br>  img.save(image_path)<br><br>  return image_path</pre><p>The definition is a bit different from the previous 2 agents because we are not using aisuite, we are using the OpenAI library directly. Let’s see what’s happening in the code:</p><ul><li>We create a prompt giving the LLM an expert role and asking it to take the rewritten essay as input and generate as output a relevant image. The prompt is stored in the <em>designer_prompt</em> var. Notice the prompt includes the rewritten report in <em>essay_summary, </em>and this is a parameter the function expects</li><li>Then we start the openai client and create the message with the following parameters: <em>model</em> (dall-e-3), <em>prompt</em> (the prompt we just created and is stored in <em>designer_prompt</em>), <em>size</em> (size of the image the model will generate), <em>image quality </em>(I set it to standard) and <em>response format</em> (set to url, meaning the response will include the url for the generated image)</li><li>We then save the image locally and return the img_path as the result of the function</li></ul><p>Now we can call the agent with the code below</p><pre>partial_essay_part = essay_part[0:3750]<br><br>#trying to extract JSON from the object<br><br>image_path = designer_agent_result = designer_agent(<br>    essay_summary=partial_essay_part,<br>)</pre><p>I had to send only a part of the essay because Dall-e only accepts up to 4,000 words, and the essay that the other agents created had around 8,000 words. We call the designer agent function, and we store the result in the <em>image_path</em> variable. The output, as shown in Figure 9 below, is:</p><ul><li>The URL for the generated image</li><li>The image name</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*St9bbvPzdJhKNKU12prUUQ.png" /><figcaption><strong>Figure 9. Graphical Designer Agent Result</strong></figcaption></figure><p>So we have an image. Let’s see how it looks with the following code</p><pre>display(ImageDisplay(filename=image_path))</pre><p>In my case, the image looks like the one in Figure 10 below. If you run the code, you most likely will get a different image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e_G3VseFObHLfl4brCCIFQ.png" /><figcaption><strong>Figure 10. Image generated by the Graphical Designer Agent</strong></figcaption></figure><p>As you can see, it’s not a perfect image (i.e., typos) but a pretty good attempt that we can use as a masthead.</p><p>OK, so we have all the outputs we have. Now we need to create the final product. For that, we create a Marketing Manager agent whose job is to give us the final report.</p><h4>Marketing Manager Agent</h4><p>The Marketing Manager Agent takes the rewritten essay and the generated image and creates a properly formatted PDF that can be shared with the Corporate Strategy Manager.</p><p>The code for the Marketing Manager is the following:</p><pre>def marketing_agent(essay:str, image_url:str, output_pdf:str):<br><br>  print(&quot;==================================&quot;)<br>  print(&quot;Marketing Manager Agent&quot;)<br>  print(&quot;==================================&quot;)<br><br>  &quot;&quot;&quot;<br>  Packages the assets into a beautifully formatted pdf report for executive review.<br><br>    Args:<br>        essay (str): the revised essay.<br>        image_url (str): URL of the essay image.<br>        output_path (str): Path to save the markdown report.<br><br>    Returns:<br>        str: Path to the saved markdown file.<br>  &quot;&quot;&quot;<br><br>  #Create a canvas<br>  c = canvas.Canvas(output_pdf, pagesize=letter)<br>  width, height = letter<br><br>  # Drawing the image first<br>  # Adjust size if needed<br>  img = ImageReader(image_path)<br>  img_width = 400  # you can adjust<br>  img_height = 250  # maintain aspect ratio if needed<br>  img_x = (width - img_width) / 2  # center horizontally<br>  img_y = height - img_height - 80  # position from top<br>  c.drawImage(img, img_x, img_y, width=img_width, height=img_height, preserveAspectRatio=True, mask=&#39;auto&#39;)<br><br><br><br>  # Add essay text<br>  c.setFont(&quot;Helvetica&quot;, 12)<br>  text_object = c.beginText(50, img_y - 40)  # Start below the image<br>  text_object.setFont(&quot;Helvetica&quot;, 12)<br>  line_height = 14<br><br>  # Split essay into lines that fit the page width<br>  max_chars_per_line = 90<br><br>  for paragraph in essay.split(&quot;\n&quot;):<br>      paragraph = paragraph.strip()<br>      if not paragraph:<br>          text_object.textLine(&quot;&quot;)<br>          continue<br><br>      # Detect Markdown headers<br>      if paragraph.startswith(&quot;### &quot;):<br>          font_name = &quot;Helvetica-Bold&quot;<br>          font_size = 14<br>          line_text = paragraph[4:].strip()<br>      elif paragraph.startswith(&quot;## &quot;):<br>          font_name = &quot;Helvetica-Bold&quot;<br>          font_size = 16<br>          line_text = paragraph[3:].strip()<br>      elif paragraph.startswith(&quot;# &quot;):<br>          font_name = &quot;Helvetica-Bold&quot;<br>          font_size = 20<br>          line_text = paragraph[2:].strip()<br>      else:<br>          font_name = &quot;Helvetica&quot;<br>          font_size = 12<br>          line_text = paragraph<br><br><br>      # Wrap text to fit page width<br>      wrapped_lines = textwrap.wrap(line_text, width=max_chars_per_line)<br>      for wline in wrapped_lines:<br>          if text_object.getY() &lt; 100:  # start new page if near bottom<br>              c.drawText(text_object)<br>              c.showPage()<br>              text_object = c.beginText(50, height - 50)<br>          text_object.setFont(font_name, font_size)<br>          text_object.textLine(wline)<br>      text_object.textLine(&quot;&quot;)  # spacing between paragraphs<br><br>  c.drawText(text_object)<br><br><br><br>  # Save PDF<br>  c.save()<br>  print(f&quot;PDF saved as {output_pdf}&quot;)</pre><p>The agent includes a bunch of code to properly format the PDF by recognizing markdown headers and by wrapping lines so they can fit on the page. Finally, we save the PDF in <em>output_pdf.</em></p><p>We can now call the marketing manager agent with the following code:</p><pre>output_pdf = &quot;report.pdf&quot;<br>essay_text = str(writeragent_result.content)<br>print(type(essay_part))<br>print(type(essay_text))<br><br>marketing_agent(essay_part, image_path, output_pdf=output_pdf)</pre><p>In this case, we are saving the PDF as report.pdf, and the output of invoking the agent is shown in Figure 11 below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yvu4Perm4oUv-z3pR_ymZA.png" /><figcaption><strong>Figure 11. Marketing Manager Agent Result</strong></figcaption></figure><p>Now we define a function to download the PDF so it can be seen in your browser.</p><pre>def display_pdf(output_pdf):<br>    # Adjust width and height as needed<br>    return IFrame(output_pdf, width=600, height=800)</pre><p>And we execute it like this:</p><pre>files.download(&quot;report.pdf&quot;)</pre><p>This will download the PDF to your computer. From there, you can open it in your browser. In my case, when I open it in the browser, I see a 7-page PDF. A couple of the pages look like Figures 12 and 13 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YNG3etVHPtZV80a-pjFywg.png" /><figcaption><strong>Figure 12. Page 1 of the Report pdf</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NcgPC0DJBZzn-G8xNsDjKQ.png" /><figcaption><strong>Figure 13. Page 2 of the Report PDF</strong></figcaption></figure><p>If you want to see the full PDF, you can see it <a href="https://drive.google.com/file/d/1kkA1jt78RRXvKAXcTVmgFQdGCTfzAyAq/view?usp=sharing">here</a>.</p><p>OK, so we are almost done with this section. Until now, you have seen how we created each agent, and we executed the task of generating this report step by step. The last step is to run the whole thing at once.</p><h3>The Whole Enchilada</h3><p>To run the whole Corporate Strategy team workflow or pipeline, we execute the following code.</p><pre>def run_create_pdf_report(output_path: str = &quot;pdf_report.md&quot;) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Runs the full corporate strategy report generation pipeline:<br>    1. Research Assistant does research and generates a report draft<br>    2. Writer reflects on the report and generates a new version<br>    3. Marketing Mangaer creates the pdf report using as input the edited report and the generated image<br>    Returns:<br>        dict: Dictionary containing all intermediate results + path to final report<br>    &quot;&quot;&quot;<br>    # 1. Run research assistant agent<br>    research_summary = research_assistant_agent(tool_functions)<br>    print(&quot;✅ Research Assistant Agent completed&quot;)<br><br>    # 2. Run writer agent<br>    essay_summary = writer_agent(<br>    research_insights=research_summary,<br>    )<br>    print(&quot;✅ Writer Agent completed&quot;)<br><br><br>    # 3. Run Graphic Designer Agent<br>    partial_essay_summary = essay_part[0:3750]<br>    image_path = designer_agent_result = designer_agent(<br>    essay_summary=partial_essay_summary,<br>    )<br>    print(&quot;✅ Graphic Designer Agent completed&quot;)<br><br><br>    # 4. Run the Marketing Manager Agent<br>    output_pdf = &quot;report.pdf&quot;<br>    essay_text = str(writeragent_result.content)<br>    marketing_agent(essay_part, image_path, output_pdf=output_pdf)<br><br>    # 5. Trigger Report Browser Download<br>    files.download(&quot;report.pdf&quot;)<br><br><br>    print(f&quot;📦 Report generated&quot;)</pre><p>You can see we create a function that invokes each agent that we created and downloads the resulting PDF to your computer. We run the whole task by just invoking the newly created function:</p><pre>report_result = run_create_pdf_report()</pre><p>and the output might look like the one below. Of course, when you run it, it will look different because the essay generated for you will most likely be different.</p><pre>==================================<br>Research Assistant Agent<br>==================================<br>&#39;Let me search for information on projected tariff impacts for 2025-2026:&#39;<br>the full dump is: <br>{&#39;choices&#39;: [&lt;aisuite.framework.choice.Choice object at 0x7c3bd2878500&gt;],<br> &#39;intermediate_responses&#39;: [&lt;aisuite.framework.chat_completion_response.ChatCompletionResponse object at 0x7c3bd2502c90&gt;,<br>                            &lt;aisuite.framework.chat_completion_response.ChatCompletionResponse object at 0x7c3bd259e210&gt;,<br>                            &lt;aisuite.framework.chat_completion_response.ChatCompletionResponse object at 0x7c3bd2540ce0&gt;,<br>                            &lt;aisuite.framework.chat_completion_response.ChatCompletionResponse object at 0x7c3bd3525eb0&gt;,<br>                            &lt;aisuite.framework.chat_completion_response.ChatCompletionResponse object at 0x7c3bd2686840&gt;],<br> &#39;usage&#39;: {&#39;completion_tokens&#39;: 90,<br>           &#39;prompt_tokens&#39;: 8614,<br>           &#39;total_tokens&#39;: 8704}}<br>✅ Research Assistant Agent completed<br>==================================<br>Writer Agent<br>==================================<br>(&#39;```json\n&#39;<br> &#39;{\n&#39;<br> &#39;  &quot;reflection&quot;: &quot;The research insights provided show that a search query was &#39;<br> &quot;made about &#39;projected economic impact of tariffs on US economy 2025-2026&#39;, &quot;<br> &#39;but no actual data or findings from this search were included in the input. &#39;<br> &#39;This indicates a significant gap in the information needed to create a &#39;<br> &#39;meaningful report on tariff impacts. Without the search results, we have no &#39;<br> &#39;specific economic projections, expert opinions, or data points to &#39;<br> &#39;incorporate into the report. A proper analysis of tariff impacts would &#39;<br> &#39;require concrete information about projected GDP effects, inflation &#39;<br> &#39;expectations, sector-specific impacts, and policy considerations for the &#39;<br> &#39;2025-2026 timeframe. Since no report draft was provided for modification, &#39;<br> &quot;I&#39;ll need to create one from scratch that acknowledges these limitations &quot;<br> &#39;while providing a framework for what such an analysis would include when the &#39;<br> &#39;proper research data becomes available.&quot;,\n&#39;<br> &#39;  &quot;rewritten_essay&quot;: &quot;# Projected Tariff Impacts on the U.S. Economy: &#39;<br> &#39;2025-2026 Outlook\\n\\n## Executive Summary\\n\\nThis report aims to analyze &#39;<br> &#39;the potential economic impacts of proposed and implemented tariffs on the &#39;<br> &#39;U.S. economy for the 2025-2026 period. While specific data projections &#39;<br> &#39;remain pending comprehensive research, this preliminary framework outlines &#39;<br> &#39;the key areas of consideration for policymakers and businesses planning for &#39;<br> &#39;potential tariff scenarios.\\n\\n## Introduction\\n\\nTariff policies &#39;<br> &#39;represent a significant tool in international trade relations with &#39;<br> &#39;far-reaching implications for domestic economic performance. Recent years &#39;<br> &#39;have witnessed substantial shifts in U.S. tariff approaches, creating both &#39;<br> &#39;challenges and opportunities across various economic sectors. Understanding &#39;<br> &#39;the projected impacts for 2025-2026 requires thorough analysis of current &#39;<br> &#39;tariff trajectories, proposed policy changes, and global trade &#39;<br> &#39;dynamics.\\n\\n## Key Areas for Assessment\\n\\n### Macroeconomic &#39;<br> &#39;Indicators\\n- GDP growth projections under various tariff scenarios\\n- &#39;<br> &#39;Inflation expectations and consumer pricing impacts\\n- Labor market &#39;<br> &#39;effects, including employment in import-competing and export-oriented &#39;<br> &#39;sectors\\n- Federal budget implications, including tariff revenue &#39;<br> &#39;projections\\n\\n### Sector-Specific Impacts\\n- Manufacturing resilience &#39;<br> &#39;and reshoring possibilities\\n- Agricultural exports and commodity &#39;<br> &#39;pricing\\n- Technology and intellectual property considerations\\n- Service &#39;<br> &#39;sector trade dependencies\\n\\n### Regional Economic Effects\\n- &#39;<br> &#39;State-by-state vulnerability assessments\\n- Urban versus rural economic &#39;<br> &#39;disparities\\n- Supply chain reorganization and geographic &#39;<br> &#39;implications\\n\\n### Global Context\\n- Retaliatory tariff scenarios and &#39;<br> &#39;bilateral trade relationships\\n- WTO compliance and multilateral trade &#39;<br> &#39;implications\\n- Currency valuation effects and capital flows\\n\\n## &#39;<br> &#39;Preliminary Recommendations\\n\\n### For Policymakers\\n- Develop &#39;<br> &#39;contingency plans for various tariff scenarios\\n- Consider targeted support &#39;<br> &#39;programs for vulnerable economic sectors\\n- Enhance coordination between &#39;<br> &#39;trade and domestic economic policy\\n\\n### For Businesses\\n- Conduct &#39;<br> &#39;supply chain resilience assessments\\n- Explore product and market &#39;<br> &#39;diversification strategies\\n- Monitor policy developments with scenario &#39;<br> &#39;planning approaches\\n\\n## Conclusion\\n\\nWhile detailed projections for &#39;<br> &#39;2025-2026 tariff impacts require comprehensive economic modeling and data &#39;<br> &#39;analysis, preparing for potential outcomes remains essential for economic &#39;<br> &#39;stability. This framework provides a starting point for stakeholders to &#39;<br> &#39;identify critical areas of concern and opportunity as tariff policies &#39;<br> &#39;evolve. Further research incorporating specific tariff rates, targeted &#39;<br> &#39;sectors, and global response scenarios will be necessary to develop &#39;<br> &#39;actionable insights for navigating this complex economic &#39;<br> &#39;landscape.\\n\\n---\\n\\n*Note: This report represents a preliminary &#39;<br> &#39;framework. Comprehensive analysis awaits detailed economic projections and &#39;<br> &#39;policy specifics for the 2025-2026 period.*&quot;\n&#39;<br> &#39;}\n&#39;<br> &#39;```&#39;)<br>the full dump is: <br>{&#39;choices&#39;: [&lt;aisuite.framework.choice.Choice object at 0x7c3bd3525970&gt;],<br> &#39;usage&#39;: {&#39;completion_tokens&#39;: 821,<br>           &#39;prompt_tokens&#39;: 236,<br>           &#39;total_tokens&#39;: 1057}}<br>content=&#39;```json\n{\n  &quot;reflection&quot;: &quot;The research insights provided show that a search query was made about \&#39;projected economic impact of tariffs on US economy 2025-2026\&#39;, but no actual data or findings from this search were included in the input. This indicates a significant gap in the information needed to create a meaningful report on tariff impacts. Without the search results, we have no specific economic projections, expert opinions, or data points to incorporate into the report. A proper analysis of tariff impacts would require concrete information about projected GDP effects, inflation expectations, sector-specific impacts, and policy considerations for the 2025-2026 timeframe. Since no report draft was provided for modification, I\&#39;ll need to create one from scratch that acknowledges these limitations while providing a framework for what such an analysis would include when the proper research data becomes available.&quot;,\n  &quot;rewritten_essay&quot;: &quot;# Projected Tariff Impacts on the U.S. Economy: 2025-2026 Outlook\\n\\n## Executive Summary\\n\\nThis report aims to analyze the potential economic impacts of proposed and implemented tariffs on the U.S. economy for the 2025-2026 period. While specific data projections remain pending comprehensive research, this preliminary framework outlines the key areas of consideration for policymakers and businesses planning for potential tariff scenarios.\\n\\n## Introduction\\n\\nTariff policies represent a significant tool in international trade relations with far-reaching implications for domestic economic performance. Recent years have witnessed substantial shifts in U.S. tariff approaches, creating both challenges and opportunities across various economic sectors. Understanding the projected impacts for 2025-2026 requires thorough analysis of current tariff trajectories, proposed policy changes, and global trade dynamics.\\n\\n## Key Areas for Assessment\\n\\n### Macroeconomic Indicators\\n- GDP growth projections under various tariff scenarios\\n- Inflation expectations and consumer pricing impacts\\n- Labor market effects, including employment in import-competing and export-oriented sectors\\n- Federal budget implications, including tariff revenue projections\\n\\n### Sector-Specific Impacts\\n- Manufacturing resilience and reshoring possibilities\\n- Agricultural exports and commodity pricing\\n- Technology and intellectual property considerations\\n- Service sector trade dependencies\\n\\n### Regional Economic Effects\\n- State-by-state vulnerability assessments\\n- Urban versus rural economic disparities\\n- Supply chain reorganization and geographic implications\\n\\n### Global Context\\n- Retaliatory tariff scenarios and bilateral trade relationships\\n- WTO compliance and multilateral trade implications\\n- Currency valuation effects and capital flows\\n\\n## Preliminary Recommendations\\n\\n### For Policymakers\\n- Develop contingency plans for various tariff scenarios\\n- Consider targeted support programs for vulnerable economic sectors\\n- Enhance coordination between trade and domestic economic policy\\n\\n### For Businesses\\n- Conduct supply chain resilience assessments\\n- Explore product and market diversification strategies\\n- Monitor policy developments with scenario planning approaches\\n\\n## Conclusion\\n\\nWhile detailed projections for 2025-2026 tariff impacts require comprehensive economic modeling and data analysis, preparing for potential outcomes remains essential for economic stability. This framework provides a starting point for stakeholders to identify critical areas of concern and opportunity as tariff policies evolve. Further research incorporating specific tariff rates, targeted sectors, and global response scenarios will be necessary to develop actionable insights for navigating this complex economic landscape.\\n\\n---\\n\\n*Note: This report represents a preliminary framework. Comprehensive analysis awaits detailed economic projections and policy specifics for the 2025-2026 period.*&quot;\n}\n```&#39; reasoning_content=None tool_calls=None role=&#39;assistant&#39; refusal=None<br>✅ Writer Agent completed<br>==================================<br>Graphical Designer Agent<br>==================================<br>https://oaidalleapiprodscus.blob.core.windows.net/private/org-8V9t7kowNWWOFm3EQIBVo9ng/user-ZhSI3N2R15uwRFf2ZmfTfVW3/img-CHe3iZEVSCXcWctE63dFwXhP.png?st=2025-10-24T06%3A27%3A11Z&amp;se=2025-10-24T08%3A27%3A11Z&amp;sp=r&amp;sv=2024-08-04&amp;sr=b&amp;rscd=inline&amp;rsct=image/png&amp;skoid=52f8f7b3-ca8d-4b21-9807-8b9df114d84c&amp;sktid=a48cca56-e6da-484e-a814-9c849652bcb3&amp;skt=2025-10-24T07%3A27%3A11Z&amp;ske=2025-10-25T07%3A27%3A11Z&amp;sks=b&amp;skv=2024-08-04&amp;sig=WhKvbocXZrUivgHObfNq8eaHs5X4Xh750Rx4FJnN4wo%3D<br>img-CHe3iZEVSCXcWctE63dFwXhP.png<br>✅ Graphic Designer Agent completed<br>==================================<br>Marketing Manager Agent<br>==================================<br>PDF saved as report.pdf<br>📦 Report generated</pre><p>and voila! The whole thing ran, and it took a couple of minutes or so. Isn’t that amazing?</p><p>I chose to create a PDF of the generated report, but you can easily generate the report in Google Docs format or Word format so you can modify it as needed. I picked PDF as the format for illustration purposes.</p><p>The code I showed can be accessed in this <a href="https://github.com/icarovazquez/agentic-ai/blob/main/Corporate_Strategy_Team.ipynb">github repo</a>.</p><h4><strong>Code Enhancements</strong></h4><p>If you are looking at the code, you probably identified a few things that can be improved. I’ll add them as time goes by; they are:</p><ul><li><em>Evals</em>: This code is broken down into steps or phases. We can look at each step, define evaluation criteria, and add it to the workflow so we can see how each agent performed</li><li><em>Different Reflection LLM</em>: right now we have the same LLM, Claude, reflecting on the output it itself created. It makes sense to use a different LLM, maybe an OpenAI one, to reflect on the output Claude generated</li><li><em>Generalization</em>: Right now, we have hardcoded a task, research the impact of tariffs on the US economy. We can modify this code to make it generic, meaning we can ask the user what research question they want answered, and we pass that question to our team of Agentic AI agents. We will go over that in a future post</li><li>Other output formats: Our final output was a PDF; we can also generate plain text or text in a particular format (Google Docs or Word)</li><li><em>Presentation/Slides</em>: In the real world, you might also be asked to do a presentation that summarizes your report findings. We will add that step in a future post.</li><li><em>Cleanup</em>: I have a bunch of print statements, mostly for debugging. We can remove those. Additionally, I have sample API calls for each tool we make available to the LLM. Those calls can be omitted if you are sure your calls are working.</li></ul><h3>The Value</h3><p>OK, so what did we learn? We learned, at a high level, what Agentic AI is, and we went through an example of what it can do. But, is there anything else we can learn? Yes, many things, in fact:</p><ul><li><strong>Productivity</strong>: One of the main values of using Agentic AI is that it helps us to be more productive. Imagine if I had to do by hand everything the LLMs, Claude and Dall-e, did for me. It would have taken me days or weeks to come up with something similar. If I were working with someone on this, I would probably have to wait for that person to be available and then wait for them to do their part. By using Agentic AI, I just saved a ton of time because it does the legwork for me</li><li><strong>A Grain of Salt</strong>: Now, in 2025, I would consider what our Agent team produced a pretty good first draft. I would take it and have a human, most likely me, enhance it by modifying the report. I could add more charts, rewrite some parts of the report, and, perhaps, add some predictions of what the economy would do with tariffs in place. I would never just take what the LLMs gave me and submit it. That’s just lazy.</li><li><strong>Limits</strong>: With some Python code and Agentic AI, we created one team that can help with one particular task: a research report. Now, imagine what other tasks we can enhance by using this technology. The sky’s the limit. We are not restricted by someone’s availability or by time. Agentic AI can help us finish our tasks faster and help us minimize the time we spend on manual tasks.</li></ul><h3>Conclusion</h3><p>Agentic AI helps us to be more productive and will automate many of our manual, time-consuming tasks in many industries. This is great because it allows us to concentrate on higher-value tasks. Instead of say, spending hours doing web searches, we can concentrate on refining the report the Agentic AI team came up with.</p><p>I encourage you to try it and use it in your day-to-day job so you can be more productive.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3aa538926e53" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Platform + MLOps = An AI Data Platform?]]></title>
            <link>https://medium.com/@icaro_vazquez/data-platform-mlops-an-ai-data-platform-46fc8338bfbb?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/46fc8338bfbb</guid>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Tue, 07 Oct 2025 18:00:21 GMT</pubDate>
            <atom:updated>2025-10-07T18:00:21.557Z</atom:updated>
            <content:encoded><![CDATA[<p>In a <a href="https://medium.com/@icaro_vazquez/what-is-a-data-platform-081549a2cef6">previous post</a>, I provided an overview of a generic Data Platform, the collection of software modules that enable enterprises and any company collecting or generating large amounts of data to collect, enrich, and utilize that data for their own purposes.</p><p>In this post, I will expand on how a Data Platform becomes an AI Data Platform by adding an MLOps Layer to its existing Data Layer. In a future post, I’ll go one step further and add a GenAI Layer to the existing Data &amp; MLOps Layers to complete what a state-of-the-art modern Data Platform should look like in the age of GenAI.</p><p>Let’s delve into how a Data Platform adds an MLOps Layer and what its components are.</p><h3>The standard Data Platform Architecture</h3><p>As a quick reminder, Figure 1 below shows a traditional Data Platform architecture diagram. For a full explanation of the architecture, please refer to the <a href="https://medium.com/@icaro_vazquez/what-is-a-data-platform-081549a2cef6">previous post.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chRg7VbMCfupRwpoiZT3Aw.png" /><figcaption><strong>Figure 1. A traditional Data Platform Architecture</strong></figcaption></figure><p>Here are the most common components:</p><ul><li><em>Data Sources:</em> These are all the different systems an enterprise uses (servers, CRM, financial systems, Cloud SaaS, etc.). All these systems generate data needed to create Data and AI-powered products, and, therefore, will be ingested into a Data Platform.</li><li><em>Data Ingestion</em>: The process of ingesting data from the various data sources, transforming it according to some business logic, and storing it in a Data Lake.</li><li><em>Destinations</em>: The sinks/data stores where data ends up. The data is, most of the time, transformed according to the requirements put forth by an engineering team that will be the user of that data. The data is then made available to consumers (i.e., downstream engineering teams) through, for example, an API.</li><li><em>Consumers</em>: There are 2 main types of consumers, shown in Figure 2 below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NryWR0iMWKEEfPFnubj7HA.png" /><figcaption><strong>Figure 2. Data Platform Consumers</strong></figcaption></figure><ul><li><em>Common Functions</em>: Lastly, 3 horizontal functions enable a Data Platform to work reliably. They are: Observability (the ability to monitor the Data Platform components to minimize downtime), Orchestration (the ability to create and schedule workflows like a data pipeline) &amp; Governance (the process that makes sure the data is available, secure &amp; usable)</li></ul><p>I refer to all the above components as the Data Layer of the Data Platform. This, for a while, was the only layer. With the advent of AI, there was a need to enable the creation of AI applications (initially ML and now GenAI), and the Data Platform was uniquely positioned to accommodate the development of these applications because it was already the enterprise data repository.</p><p>The Data Platform then added an MLOps Layer, and most enterprises today are in the process of adding a GenAIOps Layer. Now, let’s take a look at the MLOps Layer and explore the GenAIOps Layer in a future post.</p><h3>The MLOps Layer</h3><p>The MLOps Layer is the collection of functions that enable an ML Engineer to create a Machine Learning application. There are many examples of ML applications; however, my intention is not to give an exhaustive list. A simple ask to ChatGPT or a Google Search will provide comprehensive answers. In this article, I’ll mention just a couple of key examples:</p><ul><li><em>Recommendation Engines</em>: Recommendation Engines use ML to understand user preferences and provide personalized recommendations</li><li><em>Fraud Detection</em>: Enterprises use ML to identify, flag, and stop fraudulent transactions</li></ul><p>What steps does an engineer take to develop an ML application? Table 1 below gives you a list of steps and the corresponding MLOps Functionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qaJFgH4oY-lDB5GYNUVQtw.png" /><figcaption><strong>Table 1. ML Development Cycle and Corresponding MLOps Functionality</strong></figcaption></figure><p><strong>Data Wrangling</strong></p><p>Data Wrangling is the process of creating and maintaining data pipelines that make data available for ML models to consume. There are 2 modes:</p><ul><li>Ad hoc: A Data Engineering team will create and maintain the data pipeline, or an ML Engineer will create an ad hoc script to transfer the data from its current location.</li><li>Automatic: Automated Pipelines that can be triggered on a schedule or in response to an event (i.e., new data was generated). These pipelines are usually created by the ML Engineers using features provided by Cloud Providers.</li></ul><p>In my experience, most enterprises begin by relying on the Data Engineering team to create and maintain the pipelines that bring data to ML models, and usually, they do not invest in automated pipelines. ML Engineers, in general, do not get involved in creating Data Pipelines; they rely on the Data Engineering team for that. Enterprise might start investigating automated pipelines that ML Engineering can create when an opportunity arises. For example, an enterprise might be adopting a new cloud provider for MLOps, providing the opportunity to explore adding automated pipelines.</p><p><strong>Data Familiarization</strong></p><p>In the Data Familiarization step, the ML Engineer looks at the data and creates visualizations in order to better understand the data. The engineer does most of this using a notebook (<a href="https://colab.research.google.com/">Google Colab,</a> Jupyter Notebook, etc.), a language like Python, and relevant libraries like <a href="https://pandas.pydata.org/">Pandas</a>. However, they might also do ad hoc queries in different data stores to further understand the data. Finally, they might also use a Data Catalog if one is available. The Data Catalog is a foundational piece of the Data Layer of any Data Platform and, as its name suggests, contains a list of all the Data Assets an enterprise might have.</p><p>The result is that the engineer has a good understanding of the available data and can move on to the next step.</p><p><strong>Data Cleaning &amp; Processing</strong></p><p>In this step, the ML Engineer sets about cleaning the data. This means the engineer has to perform various tasks to make the data suitable for running ML model training experiments. Some of the activities the engineer performs in this step are:</p><ul><li>Remove unnecessary features: Some data columns might not be needed, so the engineer removes them from her data frame.</li><li>Modify the format of a feature(s): Some of the data columns might need to change their format. For example, if the engineer is thinking of using a Neural Network, the data used to train it needs to be in a numeric format. So any text columns will need to be reformatted. This process is called <a href="https://www.geeksforgeeks.org/machine-learning/ml-one-hot-encoding/">one-hot encoding</a>.</li><li>Get rid of/fix nulls: Some data rows might have null values, and ML models do not like that, so the engineer has to fix this problem. She can use various techniques: she can remove the columns with the missing values, add the mean/median/mode value to the missing rows, or add a constant value (say, a 0). This process is called <a href="https://www.geeksforgeeks.org/machine-learning/what-is-data-imputation/">Imputation</a>, and the 2 examples I gave are part of the Simple Imputation category.</li></ul><p>Most of this process is done in a Notebook, and the result is a dataset that can be further modified or used to train ML models and run experiments to decide which ML model will go into production.</p><p><strong>Feature Engineering</strong></p><p>In this step, the engineer takes the cleaned dataset from the previous step and further modifies it by creating additional features. Features is just a fancy way of referring to a column or a set of columns in the dataset or dataframe.</p><p>Usually, the engineer will create new features as needed based on domain knowledge. She might combine a few features in one group to create a more powerful feature or create new features from the existing data. The ML Engineer will also split the resulting dataset into 3 parts: training, test, and verification. These 3 parts of the dataset will be used to train and verify the accuracy of the model’s inferences. Inference is a fancy way of naming the model’s response.</p><p>Most of this process is done in a Notebook. Once the engineer is satisfied with the state of the dataset or set of features, she could have the option of saving this dataset in a Feature Store. A Feature Store is a repository where features are stored so they can be shared among Data Scientists, so they can be reused for future projects without having to recreate the dataset from scratch.</p><p>If your Data Platform is homegrown, it might not have a Feature Store. It could be added later once the pace of ML utilization increases in the enterprise. All cloud providers have a Feature Store in their MLOps offerings, so if your company is using a hyperscaler, chances are you already have access (or could have access) to a Feature Store.</p><p>Now the engineer is ready to move to the next step, training models.</p><p><strong>Model Experimentation</strong></p><p>By this stage, the ML Engineer already has an idea what kind of ML model they are going to use (Neural Network, Random Forest, etc.), so they pick a few models and model variations and start ‘tweaking’ the model’s hyperparameters to find the best combination.</p><p>Hyperparameters are, as the name suggests, parameters of a model that the ML Engineer can modify so the model performs better for the task at hand. For example, a Neural Network has a few hyperparameters that can be tuned: Learning Rate, number of neurons per hidden layer, number of epochs used to train the model, and so on. The process of trying out different combinations of hyperparameters is called hyperparameter optimization. If you want to learn more about it and/or see an example, you can check <a href="https://medium.com/@icaro_vazquez/neural-network-hyperparameter-optimization-with-hyperopt-f3e0cb4346c8">this article</a> I wrote about using a Python library, <a href="https://hyperopt.github.io/hyperopt/">Hyperopt</a>, to find the optimal combination of hyperparameters for a neural network.</p><p>This step is also mostly done in a Notebook.</p><p><strong>Model Training</strong></p><p>This is the process of training the ML model using the training portion of the dataset that the ML Engineer created at the end of the Feature Engineering step.</p><p>In this step, also performed in a Notebook, the training data is fed to the model using some Python code the ML Engineer writes, and the result is a trained model with a certain level of accuracy and loss.</p><p><strong>Model Evaluation</strong></p><p>The ML Engineer then decides if the model results are satisfactory. If not, she can decide to use another version of the model or to change some of the model’s hyperparameters to see if the new hyperparameters provide a better result. This process, as you might imagine, is also done in a Notebook.</p><p>In practice, the previous 3 steps are done in a kind of a loop:</p><ul><li>Pick a model (Model Experimentation)</li><li>Train it (Model Training)</li><li>Check if the results meet your criteria (Model Evaluation).</li><li>If the results are good enough, move on to the next step. If not, go back to Model Experimentation and run another experiment.</li></ul><p><strong>Model Storing</strong></p><p>Once the ML Engineer is satisfied with a model version, she can store it in a Model Store so it can be shared with other teams if needed, or it can be retrieved in case the original Model is lost. All hyperscalers provide some sort of Model Store in their MLOps offerings. If your Data Platform is homegrown, you might not have a Model Store since some teams just use a repo as a cheaper alternative.</p><p><strong>Model Packaging</strong></p><p>Now the model is ready, but it cannot be released to production in its current form since it is just code in a repo or saved in a Model Store. It needs an artifact that allows it to be consumed by other software services. So the model needs to be packaged, and this package could be:</p><ul><li>An API: the API allows another service to ask the model to make an inference when needed.</li><li>A Container: Depending on the use case, packaging the model as a container might be preferable.</li></ul><p><strong>Model Serving</strong></p><p>Model Serving is the process of releasing the ML model as an API (or a container or any other artifact) in the production environment. It involves not only releasing an API but also setting up the infrastructure behind it so it is capable of handling the expected traffic load. This usually happens using a cloud provider’s infrastructure.</p><p>It also involves setting up the model so it can be monitored for performance, accuracy, data drift, and other metrics.</p><p><strong>Model Monitoring: Observability</strong></p><p>Similar to observability in other software areas, ML model observability involves hooking up the ML model’s artifact (i.e., the API) and underlying infrastructure to an observability system like DataDog, Prometheus, Grafana, or something similar so ML engineers can track various model metrics. A bare bones list of metrics that should be tracked is:</p><ul><li>Number of API calls: How many times the model was called by other services</li><li>API Response Time: How long did it take for the API to respond</li><li>Response sent: the actual response or inference is logged so it can later be compared with the question asked, so the ML Engineer can judge if the response was correct or not.</li><li>Resource Utilization: CPU, memory, and network bandwidth utilized by the underlying infrastructure, so Site Reliability Engineers (SREs) and the ML Engineer are aware if there is a need to add more resources.</li></ul><p>In practice, the SRE, with the ML Engineer’s help, will build an observability dashboard that will contain the metrics mentioned above and many others. The whole purpose, of course, is to make sure the ML model is performing optimally.</p><p><strong>Model Monitoring: Feedback Loop</strong></p><p>Remember, we mentioned all the ML models’ inferences are logged? They are logged, among other reasons, because they can be used by the ML Engineer to decide if the ML model can be improved and/or retrained. Once an ML engineer decides a new model version needs to be promoted to production or the existing version needs to be retrained, she will use some of the logged inferences as input in retraining the model.</p><p>An engineer can use a hyperscaler’s existing feedback loop facilities or create their own. In practice, while needed, I have seen many times how, when first released to production, the feedback loop is not established with code so that it can happen as automatically as possible. Usually, engineers release an MVP, and that MVP, most of the time, does not include an automated feedback loop, so they release the API and manually perform the feedback loop. If the product or feature that uses the ML model receives a lot of attention from customers, the engineers then work to set up a more automatic feedback loop.</p><p><strong>Model Monitoring: Model Retraining</strong></p><p>In this last step of the feedback loop, the engineer will create a new dataset that includes some of the data received from the production model and will use it to retrain the model. The engineer then performs the same steps as before using a Notebook:</p><ul><li>She could change some of the model’s hyperparameters or use a different version of the model</li><li>She will retrain the model with the new dataset</li><li>She will then decide if the model is good enough for production, and either will release it or go back and try with another version of the model</li></ul><p>You will read that there are ways to automate this process, and the hyperscalers provide a way in their MLOps offerings. However, in practice, I have not seen it happen very frequently. As I mentioned before, a team might release first an MVP and, most of the time, that MVP does not include a feedback loop. This loop is then done manually by the ML Engineer, and maybe later she will look into automating it.</p><p>And that’s what an MLOps Layer looks like. Figure 2 below has a condensed Architecture that now includes the MLOps Layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uy5BnXSh0F4OQfhj2bpwFw.png" /><figcaption><strong>Figure 2. A Generic Data Platform Architecture with Data &amp; MLOps Layers</strong></figcaption></figure><h3>Conclusion</h3><p>Machine Learning use cases employ large amounts of data, so it is logical to have them anchored on the platform that holds the data that those models will use to be trained. A Data Platform that was created to handle Data use cases was then asked to enable ML use cases.</p><p>In this post, we looked at how a Data Platform adds an MLOps layer to become an AI Data Platform that enables the creation of ML products. In the next post, we will look at how the Data Platform adds a GenAIOps Layer to enable the current GenAI revolution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46fc8338bfbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is a Data Platform?]]></title>
            <link>https://medium.com/@icaro_vazquez/what-is-a-data-platform-081549a2cef6?source=rss-7d0c8410b20b------2</link>
            <guid isPermaLink="false">https://medium.com/p/081549a2cef6</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-as-a-service]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Icaro]]></dc:creator>
            <pubDate>Thu, 04 Apr 2024 04:49:36 GMT</pubDate>
            <atom:updated>2024-04-04T04:58:17.014Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>We are living in a golden era of computing with groundbreaking innovations such as the commercial internet, mobile, the cloud, and AI all coming in the past 30 years. All these innovations have helped us in many ways to be more connected and more productive and all of them have one thing in common: they have fueled an explosion of data that is produced at staggering amounts every day. For today’s enterprises, this data can be used to gain insights about their users, about how their products are used, and to gain a competitive edge against the competition, among other benefits.</p><p>Managing the torrent of data generated is one enterprise challenge that can be very expensive, labor-intensive, and complex. Organizations today do not have the luxury of not managing their data and need an optimized solution to take advantage of all the data generated by their users and by the enterprise itself. This is where a <a href="https://en.wikipedia.org/wiki/Data_platform">Data Platform</a> comes in.</p><p>In this article, we will provide an overview of what a<strong> </strong>Data Platform<strong> </strong>is, the main components of a modern Data Platform<strong>,</strong> the benefits of using a Data Platform, and a few insights from our time selling and building Data Platforms for different Silicon Valley companies.</p><h3>What is a Data Platform?</h3><p>There are many definitions of a Data Platform but we are going to concentrate on three. A Data Platform is:</p><ul><li>A software suite that enables organizations to process and manage large amounts of data quickly and efficiently. It manages data collection, normalization, transformation, and application for a given data product.</li><li>A central repository and processing house for all the organization’s data.</li><li>A collection of data pipelines that ingest data, store it, transform it, and deliver it to a variety of destinations so the data can be consumed by different enterprise groups for their purposes.</li></ul><p>In layman’s terms, it is a collection of software tools that allow the organization to move and transform data from one place to another so it can be used for various purposes.</p><h3>Who is responsible for the Data Platform?</h3><p>This will change from enterprise to enterprise but, in general, there is one engineering group responsible for the overall Data Platform. It could be called the Data Platform team, the Data Engineering team, the Data team, etc. This team is composed of the following individuals shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*EdSo615lnNEv1aoX" /></figure><h3>Who are the customers of the Data Platform?</h3><p>Typically, the customers of the Data Platform are other engineering groups within the organization. Sometimes, external partners might be given access to certain parts of the Data Platform, i.e. a data warehouse. The internal engineering teams either have data that needs to be ingested into the Data Platform or need to get some data out of the Data Platform. In this day and age, most engineering teams own one or more services that are available to other engineering teams within the enterprise. As such, there is always a need to get data from other teams to use in your service. For example, an ML group that is say, in charge of identifying threats that come to the enterprise’s customer-facing service will need data from say the network team, the metadata team, and other teams to have all the data it needs to train its models. This data is delivered to the ML team through the Data Platform.</p><p>Now let’s take a look at a typical modern Data Platform Architecture.</p><h3>What are the components of a Data Platform?</h3><p>A sample of modern Data Platform architecture is shown below. The main components are Data Sources, Data Ingestion, Destinations, and the Analytics Layer but there are more components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*ci3duvI52o5hH5CP1nnJqA.jpeg" /></figure><p>Let’s briefly go over each of the components of the architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*DqpYNcPKwhEh26SAjoorfg.png" /></figure><p>An organization’s data is scattered around many different systems (CRM system, web server logs, financial systems, cloud provider logs, etc.) and most of this data will need to be ingested by the Data Platform. It is the job of the Data Platform team and the engineering owner of the data source to work together so the relevant data can be ingested by the Data Platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*Y6aDy7E7-VjJZLJm5Cfdpg.png" /></figure><p>The process of extracting data from one system and delivering it to another. The source systems can have structured or unstructured data so the Data Platform team is left with the task of figuring out how to get the data out from all these systems. This task is usually referred to as Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT). These terms are explained below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*9lxqVYCw-VDZKU3fDTMJTA.jpeg" /></figure><p>Having said that, there are a couple of ways of processing data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*jZSywj50mpcpPkrLrQKjLA.jpeg" /></figure><p>The most common software choices for Batch Processing are <a href="https://www.fivetran.com/">FiveTran</a> and <a href="https://www.stitchdata.com/">Stitch</a> and the most common tools for Stream Processing are <a href="https://kafka.apache.org/">Kafka</a>, <a href="https://cloud.google.com/pubsub?hl=en">Google Pub/Sub</a>, and <a href="https://spark.apache.org/">Apache Spark</a>. I will also mention Eventing since some people make the case that Eventing should stand on its own. It is a type of stream processing and occurs when there is a need throughout the organization to surface information in real-time or close to real-time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*5ygNZua4M1yc39sy6qKKnA.png" /></figure><p>In this step, data is cleaned using business logic so it can be ready for analysis and reporting. In the old days, this was done before loading the data in a database or data warehouse due to costs. These days the data can be loaded into a data lake and transformed later. Popular tools here are <a href="https://www.getdbt.com/">dbt</a>, <a href="https://cloud.google.com/dataform?hl=en">Dataform</a>, and custom code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*kCTGsH3b_G6iWVuNGupyRA.png" /></figure><p>There are 2 main types of destinations</p><ul><li><em>Sinks/Data Warehouses</em>: Before or after the data is transformed it needs to be stored somewhere. In more than a few cases, the data is sent to ‘sinks’ where it is stored. One example of a sink is a data warehouse. The data warehouse is a type of database where the data has already been transformed/optimized for reporting. Without trying to spend too much time on database technology, let’s do a quick definition. In general, there are 2 types of database applications shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*932rF8GQeJ0-Qdwq" /></figure><ul><li><em>Services/Applications</em>: The organization will have different groups that need to be notified when a new event has been generated and they need to react immediately to that event. These groups will ‘subscribe’ to the stream of events that are generated by the enterprise’s users, partners, or other internal groups, and will need to take action every time an event is generated. As we already mentioned, this is a real-time case so the events generated are streamed so that all subscribed services can consume the event and take the appropriate action for that event. For example, consider the case of a user who uploads a file to a cloud content service. The cloud content service’s Data Platform will need to generate an event for that file upload and multiple groups within the enterprise will need to consume that event. One group, let’s call it the file system group, will consume the event and generate different image icons of the file uploaded so those icons can be used in the desktop and mobile versions of the cloud content service app.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*abw3b8W7dlNili8Wso7RZQ.png" /></figure><p>This layer is the one that includes the business intelligence and visualization tools that generate reports/charts that multiple enterprise groups will consume. This is the layer where non-technical users will interact with the data. Common groups that access this layer are Business Analysts, Financial Analysts, Engineering Managers, Executives, etc. Popular tools here are <a href="https://www.tableau.com/">Tableau</a>, <a href="https://www.microsoft.com/en-us/power-platform/products/power-bi">PowerBI</a>, and <a href="https://cloud.google.com/looker?hl=en">Looker</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*lst9m2udA9M321jbgRvvGg.png" /></figure><p>Orchestration means automating multiple tasks into a single process. Orchestration software enables data engineers to create workflows that trigger jobs that ingest, transform, and deliver data to the appropriate sources at the appropriate time. An example of a popular tool to use here is Apache Airflow, either the open-source version, the cloud-branded version, or another commercial version available such as Astronomer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*-lqhrPb6hIXG9Gw-TR0hHQ.png" /></figure><p>The Data Platform is a critical system in an enterprise and, as such, needs to minimize downtime and be monitored at all times. The Data Platform usually does this through an observability tool that allows it to understand the health of the data platform at all times. The observability solution needs to look at a high level, at 2 levels:</p><ul><li>The health of the Data Platform itself.</li><li>The health of the underlying infrastructure that powers the Data Platform.</li></ul><p>Some of the most important things an observability solution will need to cover are:</p><ul><li>Freshness: Is the data recent? When was the last time data was ingested? When was the last time data was written to a sink/service?</li><li>Data Arrival: Has all the data arrived at its destination?</li><li>Lineage: For a given data asset, what are the upstream and downstream sources that are impacted by it?</li></ul><p>Some popular observability solutions here are <a href="https://tanzu.vmware.com/observability">VMWare Tanzu</a>, <a href="https://www.datadoghq.com/">DataDog</a>, <a href="https://www.dynatrace.com/">Dynatrace</a>, and <a href="https://newrelic.com/">NewRelic</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*R8-4XkHoMbvXSfpX-VP6VQ.png" /></figure><p>This is the process that makes sure the data is available, secure, and usable. Some common questions this layer answers are:</p><ul><li>Who should have access to the data?</li><li>Who accessed the data and when?</li><li>Where is the data stored?</li><li>When is data retired?</li></ul><p>Governance is not just software. It’s rather a combination of software, processes, principles, and people. One software tool that can help here is a Data Catalog. A Data Catalog is a detailed inventory of all the data assets in an organization and uses metadata to describe these assets. One of the main use cases for governance is compliance. Organizations in many industries such as finance or health care have to comply with regulations such as HIPPA and they need to demonstrate how their data is secured, who accessed it, when was accessed, and so on. A Data Catalog is a foundational piece since it contains the list of all the data assets the enterprise owns.</p><h3>How important is the Data Platform?</h3><p>We wanted to touch upon this subject because we find that it is overlooked many times. We already established the majority of the Data Platform customers are internal engineering teams. These teams own one or more services and many of these services are critical for the well-being of the product the enterprise’s customers buy. In other words, these services are critical to generate revenue. As such, their main focus is to release features that are asked by end customers and these asks are reflected in the engineering teams’ roadmap.</p><p>This fact brings an important challenge for the Data Platform that we want to highlight. The Data Platform team will have asks for most engineering teams in a company and these asks very rarely, rarely, will be at the top of the priority list for the internal engineering teams since the Data Platform requests, i.e., help us write some code to ingest data from your service, will not directly generate revenue. Furthermore, in a big enterprise, the Data Platform team will need data from engineering teams in completely different business units or divisions. This additional organizational layer makes the challenge even more complicated. So the Data Platform team, most likely, the Data Platform Product Managers, have an interesting problem: how do we get other engineering teams to pay attention and act on our requests when these asks are nowhere near the top of the priority list for a given engineering team?</p><p>We have dealt with this problem before and wanted to share a few strategies that have helped us with this issue:</p><ul><li><em>Understand your customers’ use cases:</em> Because of its nature, the Data Platform has the closest thing an enterprise has to a 360-degree view of all the data generated and its members should understand the use cases of all their customers. So one of the first things a Data Platform Product Manager should do is understand all their customers’ use cases so they can see what kind of data they generate and what kind of data they might need. This immersive view should allow the Product Manager to start connecting the dots between the different engineering teams and, for example, think about how data from Team A can enhance Team B’s service. An additional benefit of this exercise is it will also help the Data Platform Product Manager to plan their roadmap.</li><li><em>Show value:</em> Once the customer use cases are understood the next step is to show the value the Data Platform can bring to the use case that the engineering team is supporting. Since the Data Platform has access to most of the data generated by the enterprise the Data Platform Product guys are in a prime position to show how the data they have can enhance the other product manager’s service. For example, additional data given to an ML team might end up as part of the dataset that is used to train some of their models and, as a result, the accuracy of the model might increase, which results in a better user experience and, hopefully, more revenue. Another example would be how the data the Data Platform has can help uncover outages, possible performance delays, or other performance issues that impact the service’s availability. Our point is that, most likely, there is data that can help the engineering team’s service and we should be able to show that to the other engineering teams. This will increase their interest in the Data Platform and will make them more amenable to receiving that data and, hopefully, give more priority to our asks. It’s a ‘I scratch your back, you scratch my back’ kind of situation.</li><li><em>Plan early, way early:</em> Since every engineering team has a large list of requests/features in their roadmap, get ahead of the game and try to get into the other team’s roadmap ahead of schedule. You most likely will negotiate with the other engineering team’s product manager. That product manager probably has a full roadmap for the current quarter or so; therefore start the conversation at least one or two quarters early so you can get a spot on that engineering team’s roadmap for the next quarter.</li><li><em>Establish relationships with the critical players</em>: This might sound silly to some but never underestimate the importance of establishing relationships with other product managers, engineering managers, and so on. People like to work with people they can trust, who are honest, transparent, and reliable so spend the time getting to know the important players and meet them frequently not only to keep updated on what are they doing but also to get to know them better and to update them on what is the vision for the Data Platform and why it is important to them. Finally, when you say you are going to deliver something for them, deliver it.</li></ul><p>The actions above have helped us to get buy-in from other engineering teams so we can get our asks in their roadmap and we can make the Data Platform and the other engineering services better.</p><h3>But… what about ArtificiaI Intelligence (AI)?</h3><p>This article would not be complete without mentioning AI and Generative AI. Perhaps an example here will help us to kill 2 birds with one stone: we will use a slightly changed example from real life and it applies to ML, which includes Generative AI. People also asked us over time why is the Data Platform relevant to AI use cases so we hope to answer both questions with this example.</p><p>Assume you are part of a Machine Learning group that aims to create Machine Learning (ML) models that will recognize threats (bad actors trying to take over user accounts, bad actors that have compromised accounts and are doing unusual actions using those accounts, etc.). The Data Platform Product Manager asks for a meeting and you roll your eyes and think ‘Another meeting, why should I meet with people I don’t have anything to do with?’ Well, let me tell you why.</p><p>A Data Platform team’s connection with an ML group is composed of the following dimensions:</p><ul><li><em>Notebooks Instance</em>: Data Scientists use notebooks (i.e., Jupyter Notebooks) to clean up their datasets, create their models, and train them. A Data Platform team, on many occasions, is the one that creates, supports, and provides the Notebooks instance to the ML teams in the company.</li><li><em>Access to Data</em>: ML models need large amounts of data to be trained and Data Scientists spend a lot of time cleaning this data and formatting it for their purposes. Guess where the majority of this data is stored? Yes, the Data Platform. The data could be directly in the Data Lake or one of the many data sinks the Data Platform can have. So Data Scientists will need to access either the Data Lake or the Data Warehouses.</li></ul><p>These two samples show how the work of the ML team is enabled by the Data Platform and why the AI team should care about the Data Platform. It is crucial for the success of the ML team.</p><p>Additionally, you might have heard the demand for GPUs is through the roof since Large Language Models (LLMs) are big and need to be trained using multiple GPUs so cloud providers and big enterprises are gobbling up these GPUs. However, a kind of forgotten point is that the LLMs are trained with huge datasets and these datasets will need to be stored somewhere, and that somewhere is, you guessed it, the Data Platform.</p><h3>Anything else?</h3><p>We have worked in Silicon Valley companies that sell software that is part of a Data Platform and as part of groups that build Data Platforms within a company so we wanted to mention a few of the things we have observed and learned in our time in this fascinating space:</p><ul><li><em>The Data Platform grew organically</em>: This is a polite way of saying the company didn’t have a Data Platform group responsible for offering Data as a Service so different engineering teams created their own data pipelines over time. As the company grew the need for having data available at all times necessitated the creation of a Data Platform team that takes over responsibility for all the existing data pipelines. We have seen this case in enterprises that were born before the cloud and that grew over time and didn’t invest initially in a Data Platform team. Once the Data Platform team is created, the original creators of the existing pipelines are gone and the Data Platform team is saddled with supporting and enhancing data pipelines they may not understand but cannot shut down. Over time, the team will migrate the Data Platform from data centers to the cloud so it can take advantage of modern-day software benefits such as orchestration, auto-scaling, etc.</li><li><em>This is a maze of software tools</em>: There is no single vendor that can provide all the pieces of a Data Platform so the Data Platform engineers have to use and manage many different tools that must work together to move, transform, and deliver data.</li><li><em>The software vendor is not helping</em>: Related to the previous point, many times we have seen how a vendor providing a piece of software for a data platform cannot replicate a customer use case due to many constraints. For example, replicating the size of a Data Platform a customer has could be cost-prohibiting for the software vendor. In such cases, the software vendor is not going to be of much help, and, in reality, the enterprise is testing the vendor software in their environment. Both parties are learning together at the same time. This is not optimal but a somewhat common occurrence.</li><li><em>The observability solution is not helping</em>: A Data Platform has to be available 24/7 because, among many reasons, many long-running jobs will run at night. This means that if the night workload fails, the on-call Data Platform engineers will need to wake up and try to figure out what is the problem and try to fix it. Sadly, observability is still in the Stoneage in this regard. In all our time working in Silicon Valley, the developer experience of observability tools has not changed much. If you look at a developer in 2000 using an observability solution and a developer in 2024 using an observability solution you will see the same screen: a bunch of dashboards with tons of charts that show information such as response time, memory usage, CPU usage, etc. It is up to the developer to correlate one chart to another to figure out where the problem is. Good luck with that!</li></ul><p>This list is just a partial list of the lessons we have incurred during our time building Data Platforms and we hope they can be useful to you too.</p><h3>Conclusion</h3><p>Data will continue to explode as more and more applications are created and people use them so we can expect the importance of modern Data Platforms to continue to increase as well. As Generative AI continues to explode in popularity and more Gen AI-based applications are created, the Data Platform will be critical for the success of these applications, even as these applications add more requirements to the Data Platform team.</p><p>Organizations do not have the luxury of ignoring the importance of using data as a competitive advantage and the Data Platform is a prime enabler of this desire. Because of this, we expect Data Platforms to continue to evolve, and provide a better, less complex experience for their users while continuing to grow in size to keep up with the amount of data modern-day organizations are generating.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=081549a2cef6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>