<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Santhilata KV on Medium]]></title>
        <description><![CDATA[Stories by Santhilata KV on Medium]]></description>
        <link>https://medium.com/@santhilatakv?source=rss-444748761ef8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*BegDtF7H7NaKd9Qh.</url>
            <title>Stories by Santhilata KV on Medium</title>
            <link>https://medium.com/@santhilatakv?source=rss-444748761ef8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 00:35:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@santhilatakv/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to read a frontier model’s safety report]]></title>
            <link>https://santhilatakv.medium.com/how-to-read-a-frontier-models-safety-report-e0846406965e?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0846406965e</guid>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[ai-frontiers]]></category>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Sat, 14 Nov 2026 15:15:16 GMT</pubDate>
            <atom:updated>2026-11-14T15:15:16.929Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J2kLTPxMTYV-NY2Q" /><figcaption>Photo by <a href="https://unsplash.com/@joetography?utm_source=medium&amp;utm_medium=referral">Joe Dudeck</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Every major AI release now comes with a long PDF called a system card. This is the same card that AI Governance practices insist you use for every model that you deploy. Most people never open one. That’s a shame, because it’s the closest thing we have to an instruction manual for the risks.</p><p>One real example: Anthropic’s system card for Claude Fable 5.1 and Claude Mythos 5.1, published September 1, 2026. It runs to 212 pages. I read the executive summary (just like anybody else!) and the start of the main text, so this is a quick guide to the first pass, not more.</p><p>I have taken Claude’s help to decode several labels for me while collating the information for this post. It is to establish the basics to create system cards — what information they should contain for auditing and why.</p><p><strong>Step 1: Check what you’re actually getting</strong></p><p>The first thing to find is which version you can use. The card covers two configurations of the same weights. Fable 5.1 is the general-access version, with safeguards that block some high-risk biology and cybersecurity requests. Mythos 5.1 has the same weights with looser safeguards, and direct access is limited to vetted users. Two versions, two risk profiles. Read the one you would deploy.</p><p><strong>Step 2: Where to find the risk ratings and what to do with them?</strong></p><p>Safety cards use a small set of labels. Here is what the Fable 5.1 summary says, in plain terms:</p><ul><li>Chemical and biological risk is rated CB-1. The authors say the model falls short of the next level, CB-2, which would mean it can replace rare expert talent. They hold that judgement with some uncertainty, so the same biological safeguards stay in place.</li><li>Autonomous AI research risk is rated low. Testing by an outside group, METR, was consistent with that.</li><li>Alignment risk, meaning the chance the model pursues goals its developers did not intend, was raised from very low to low. The authors cite uncertainty after recent cybersecurity evaluation incidents.</li></ul><p>If a label moves, ask why. This one moved because of incident reports, not because a new test showed danger. That is worth knowing.</p><p><strong>Step 3: Read the cyber section as a warning</strong></p><p>The card says Mythos 5.1 has the strongest cyber capabilities of any model the company has released. The authors also say they have not found evidence of a critical-severity jailbreak for Fable 5.1. Both claims are in the summary. Neither proves the model is safe. They tell you where the company thinks the risk is.</p><p><strong>Step 4: Look for the trade-offs. What about costs?</strong></p><p>Good cards admit costs. Over-refusals were rare, but single-turn harmful requests got undesirable answers somewhat more often than in recent Claude models. The cyber classifiers also block some harmless requests and fire more often than those on Opus 5.</p><p>That’s a pattern to watch for across every vendor. Being safer on one measure can mean being worse on another.</p><p><strong>Step 5: Read the incidents</strong></p><p>This is the part most readers skip, and it’s the most useful. The card reports that, in internal monitoring, the model worked around safety classifiers or broken permission hooks in fewer than 0.01% of completions, sometimes by overstating what the user had authorised. In one external test, a partner saw the model exploit a sandbox weakness to read files outside its environment. The authors rated it low severity and reported it for transparency.</p><p>Low rates still matter at scale. 0.1% of a million is still a real number that could give you tears!</p><p><strong>Step 6: Check the caveats</strong></p><p>The authors list their own doubts. Automated biology tests are not reliable proxies for real-world danger. Some benchmarks are saturated. The alignment evidence is weaker because the model has had less internal use. The card also says the model is among the most capable at controlling its own extended thinking, which the authors treat as weak evidence that it may be harder to monitor.</p><p>A card that lists its limits is easier to trust than one that doesn’t.</p><p><strong>What this card doesn’t tell you</strong></p><p>It’s written by the company that built the model. It is a self-assessment ( often this is the case! ), and independent reviewers have flagged the same point about other vendors’ cards. Use it as a starting point, then check the claims that matter to you against outside evaluations.</p><p><strong>The short checklist</strong></p><p>Find which version you can use. Find the risk ratings and ask what moved them. Read the incidents. Look for trade-offs. Read the caveats. Then decide what you still need to test yourself.</p><p>In this series, we started with what frontier AI means, went through the legal lines, what it means for jobs, and how to read a company’s own safety paperwork. If you read only one of these posts, make it this one. The next model release will come with a new card, and you’ll be able to check it.</p><p><em>References:</em></p><ul><li>Anthropic, “System Card: Claude Fable 5.1 &amp; Claude Mythos 5.1” (Sept 1, 2026). <a href="https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&amp;%20Claude%20Mythos%205.1%20System%20Card.pdf">https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&amp;%20Claude%20Mythos%205.1%20System%20Card.pdf</a></li><li>TechJack Solutions, “Claude Fable 5.1 System Card Breakdown” (Sept 1, 2026), secondary commentary. <a href="https://techjacksolutions.com/?p=83808">https://techjacksolutions.com/?p=83808</a></li><li>Anthropic Claude Opus 5 and Fable 5 system card coverage, for context on the earlier cards cited in the commentary.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0846406965e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What frontier AI means for your job]]></title>
            <link>https://santhilatakv.medium.com/what-frontier-ai-means-for-your-job-cfb34e829b37?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfb34e829b37</guid>
            <category><![CDATA[ai-frontiers]]></category>
            <category><![CDATA[job-redundancy]]></category>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Thu, 12 Nov 2026 17:28:33 GMT</pubDate>
            <atom:updated>2026-11-12T17:36:28.499Z</atom:updated>
            <content:encoded><![CDATA[<p>with examples</p><p>We understood what frontier AI is in my previous <a href="https://santhilatakv.medium.com/what-on-earth-is-frontier-ai-3098682a5a5f">post1</a> and <a href="https://santhilatakv.medium.com/why-frontier-depends-on-the-law-612a7b40a0be?postPublishedType=repub">post2</a>. Let us see what frontier AI means for our jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NfTATQuB-CQ4Ke0n" /><figcaption>Photo by <a href="https://unsplash.com/@silverkblack?utm_source=medium&amp;utm_medium=referral">Vitaly Gariev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>A Scenario</strong></p><p>Every few months, someone posts a chart claiming a job will vanish by next year. I would rather look at the work itself. Take a finance team. Which tasks have changed, which haven’t, and what does that mean for the person doing them?</p><p>Start with what the latest models can do. Anthropic’s release notes list Claude Fable 5.1 and Mythos 5.1 launching on September 1, 2026, and Claude Opus 5.5 on September 22. OpenAI’s notes show GPT-6.1 Terra arriving on October 26, which it describes as supporting visual reasoning, computer use, and writing. Computer use matters here. It means a model can work through the same screens an analyst uses, not only the text they paste in.</p><p>Now the limit. A benchmark called FrontierFinance tested real investment research tasks. No frontier model, including Anthropic’s Claude Fable 5, OpenAI’s GPT-5.6 Sol, and Google’s Gemini 3.1 Pro, clears 50% accuracy on real investment research tasks, with the top score at just 49.2%.</p><p>T<em>hat test was run by a vendor and covers one kind of work, so it’s a signal, not a verdict. It still tells you that hard analysis needs a human check.</em></p><p>Here are three examples. The figures are invented for illustration. The tasks are typical.</p><p><strong>Example 1: Month-end variance commentary.</strong> A model reads the ledger export and drafts one paragraph per cost centre explaining movements. The analyst checks each number against the source. Work that took a day now takes an hour, and the analyst’s job becomes checking, not typing.</p><p><strong>Example 2: Expense claim triage.</strong> A model flags claims that look duplicated or out of policy. A person decides what to do with each flag. The model does the sorting. The person keeps the judgement and the accountability.</p><p><strong>Example 3: Reading a long annual report.</strong> A model summarises a 200-page report and pulls the revenue line. Most of the time, that works. But suppose a note near the back restates last year’s figure. A summary that skips the note reads smoothly and is wrong. This is the failure to watch for: fluent output that quietly misses one important detail.</p><p>So what changes? Routine preparation shrinks. A person can hand off drafting, reconciling, and first-pass reading to a model, then review the output. The skills that gain value are defining the question clearly, knowing which sources matter, and recognising when an answer looks right but isn’t. Governance work also grows, because someone has to set rules for how these tools are used and who signs off on them.</p><p>The regulators seem to agree on that last point. In July 2026, the EU’s banking, insurance and markets supervisors told firms to cover frontier-AI cyber threats in their governance and resilience work. That source is an aggregator, so check the original statement before quoting it.</p><p>The harder question is who is accountable when the model gets example 3 wrong. The analyst who trusted the summary? The manager who approved the workflow? The vendor? That is the next post.</p><p><em>References:</em></p><ul><li>Anthropic, Claude Help Center release notes. <a href="https://support.claude.com/en/articles/12138966-release-notes">https://support.claude.com/en/articles/12138966-release-notes</a></li><li>OpenAI, Model Release Notes. <a href="https://help.openai.com/en/articles/9624314-model-release-notes">https://help.openai.com/en/articles/9624314-model-release-notes</a></li><li>TechTimes, coverage of the FrontierFinance benchmark (Jul 2026). <a href="https://www.techtimes.com/articles/322087/20260729/finance-ai-ceiling-claude-fable-5-tops-frontier-models-just-49-investment-tasks.htm">https://www.techtimes.com/articles/322087/20260729/finance-ai-ceiling-claude-fable-5-tops-frontier-models-just-49-investment-tasks.htm</a></li><li>Wheresthe.ai, frontier AI timeline (regulatory statements). <a href="https://wheresthe.ai/t/frontier-models/">https://wheresthe.ai/t/frontier-models/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfb34e829b37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why “frontier” depends on the law]]></title>
            <link>https://santhilatakv.medium.com/why-frontier-depends-on-the-law-612a7b40a0be?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/612a7b40a0be</guid>
            <category><![CDATA[ai-frontiers]]></category>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 16:21:08 GMT</pubDate>
            <atom:updated>2026-11-11T12:10:29.609Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Same model, different verdict</strong></p><p>Following my previous <a href="https://santhilatakv.medium.com/what-on-earth-is-frontier-ai-3098682a5a5f">post</a> on frontier AI, let us see how the model and interpretation change with the law.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OHyrYGz8afZCeFBX" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Picture a fitness test. One person passes the sprint. The same person fails the rope climb. Overall, nobody thinks they’re fit or unfit. The test just measures one thing. Frontier AI works a bit like that. There isn’t one official test for it. Researchers and lawmakers choose different ways to measure, and each way gives a different answer.</p><p>The CASRAI guide (see reference) breaks the measure into three common approaches. One counts how much computing power went into training. Another looks at what the model can actually do. A third looks at the risks it might carry. None of them is the official one. They were written for different jobs, so a model can be frontier under one rule and ordinary under another.</p><p>Let&#39;s see how that plays out in practice. The EU presumes a model is high-impact once its training crosses 10²⁵ FLOPs, though the company can argue the presumption does not hold. California’s SB 53 sets its line at 10²⁶. So a model trained with 10²⁵ operations lands in the EU’s high-impact group but sits below California’s threshold. Same model. Different answer, depending on which country’s border you’re standing at.</p><p>Why does this matter to us? For a company, the label can decide which rules to follow, what to report, and what can be fined for. For a reader who follows policy news, it explains why two headlines can call the same system “frontier” and “not frontier” on the same day. Both can be correct.</p><p>But it also raises a bigger question. Compute is easy to count. How do you pin down the capability? Very hard. Risk is the hardest of all. So which measure should a law use? And if the answer changes every few years, can a fixed number ever keep up?</p><p><strong>So should the definition of “frontier AI” be fixed or flexible?</strong></p><p>In my view, lawmakers need a way to decide which AI models face extra rules. The obvious measure is a quantifiable number. A number also resists political pressure, because nobody has to argue about whether a model “feels” dangerous. Count the computing power used to train a model, and if it passes a line, the strict rules apply.</p><p>A compute line is easy to check. A company can add up the training hours and know where it stands. Regulators can verify it too. Fixed rules also give businesses time to plan. The compute tracks risk reasonably well. Bigger training runs usually mean more capable models.</p><p>The trouble is that the number may not stay useful for long. Fixed compute thresholds for “frontier” AI models can quickly become outdated, and policymakers should adopt a dynamic, capability-based definition that keeps regulation focused on models posing truly exceptional risks. A model trained with less compute could be just as capable, and a rule keyed to compute would miss it. An older, clumsy model trained with a lot of compute could still get flagged.</p><p><strong>The middle position</strong></p><p>The Georgetown Center for Security and Emerging Technology (CSET) found that compute thresholds could play a useful role in regulatory definitions, but are unlikely to work as a complete solution. In other words, compute can be a useful first filter, not the whole test.</p><p>The best argument against a flexible definition is legal certainty. If the test depends on what a model can do, a company may not know whether it is covered until a regulator decides. That creates uncertainty, and uncertainty can slow down responsible developers more than reckless ones.</p><p>Process is part of the problem. Defining frontier AI through a formal rulemaking takes time. One legal analysis notes that by default the rulemaking process is time-consuming. The same analysis warns that a broad agency definition could face legal challenge if Congress has not authorised it. A capability test that changes often would need to keep passing that process, which is hard to do quickly. So the question is, how can a rule stay current without becoming so vague that nobody can tell if they must comply?</p><p><em>References:</em></p><ul><li>CASRAI, “What is a frontier AI model?” <a href="https://casrai.org/guides/what-is-a-frontier-ai-model">https://casrai.org/guides/what-is-a-frontier-ai-model</a></li><li>EU AI Act, GPAI guidelines overview. <a href="https://artificialintelligenceact.eu/gpai-guidelines-overview/">https://artificialintelligenceact.eu/gpai-guidelines-overview/</a></li><li>Future of Privacy Forum, “California’s SB 53: The First Frontier AI Law, Explained.” <a href="https://fpf.org/blog/californias-sb-53-the-first-frontier-ai-law-explained/">https://fpf.org/blog/californias-sb-53-the-first-frontier-ai-law-explained/</a></li><li>ITIF / Data Innovation, “AI’s Frontier Is Moving. Its Legal Definition Should Too” (Aug 2026). <a href="https://itif.org/publications/2026/08/28/ais-frontier-is-moving-its-legal-definition-should-too/">https://itif.org/publications/2026/08/28/ais-frontier-is-moving-its-legal-definition-should-too/</a></li><li>CSET, Georgetown University, on defining frontier AI for regulation. <a href="https://cset.georgetown.edu/?p=16031">https://cset.georgetown.edu/?p=16031</a></li><li>Law and AI, on frontier AI definitions in US regulation. <a href="https://law-ai.org/frontier-ai-definitions/">https://law-ai.org/frontier-ai-definitions/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=612a7b40a0be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What on earth is “frontier AI”?]]></title>
            <link>https://santhilatakv.medium.com/what-on-earth-is-frontier-ai-3098682a5a5f?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3098682a5a5f</guid>
            <category><![CDATA[ai-frontiers]]></category>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 11:48:39 GMT</pubDate>
            <atom:updated>2026-11-10T16:27:45.955Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>And why does the answer keep changing??</strong></h3><p>You’ve probably seen the phrase <strong>Frontier AI</strong> in news stories about governments and AI companies. Does it sound like something from a sci-fi film? Actually, it isn’t. The idea is simple. The target keeps moving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R4iyldXhXDw0Hfnv" /><figcaption>Photo by <a href="https://unsplash.com/@pf91_photography?utm_source=medium&amp;utm_medium=referral">Patrick</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>My version says, Frontier AI means the most capable AI available right now. Imagine a group of hikers crossing a foggy field. The people at the very front are the frontier. Everyone behind them is still following. The front moves every time someone takes a step, so “the frontier” always means wherever the leading edge is today.</p><p>Yesterday’s frontier becomes today’s ordinary tool. Nobody’s last year’s phone stays “the latest phone” for long. AI works the same way, only faster. Experts usually describe frontier systems with three traits. They sit near the current top of the field. They are highly capable. And they can handle many kinds of tasks, not just one. Think of a Swiss Army knife, just not a corkscrew.</p><p>Here comes the twist. Since there isn’t an official definition, different groups draw the line differently. One approach counts how much computing power went into training a model. Another looks at what the model can actually do. A third looks at the risks it might pose. A single model can be frontier under one rule and not under another. It’s like a sign at a funfair ride that says “You must be this tall.” The sign doesn’t care how impressive you are. It cares about one measurement.</p><p>That measurement matters more than it sounds. In some laws, crossing the line switches on legal duties for the companies that build these models. So “frontier” isn’t just a vibe or a tech-industry buzzword. It can decide who has to follow strict rules.</p><p><strong>Let me explain with an example. One phrase and three legal answers</strong></p><p>The EU, California, and New York all regulate “frontier” AI, but each draws the line at a different position.</p><p>The EU presumes a model poses systemic risk once it is trained with at least 10²⁵ FLOPs. Providers may contest the systemic risk classification by supplying robust <a href="https://artificialintelligenceact.eu/gpai-guidelines-overview/">evidence</a>. (see references)</p><p>California’s SB 53, signed into law in 2025, sets a tenfold higher bar of more than 10²⁶ operations. Its strictest tier applies to developers with over $500M in annual revenue. Penalties are capped at $1 million per <a href="https://www.regulatoryoversight.com/2025/10/california-charts-the-frontier-with-first-law-setting-reporting-and-compliance-requirements-for-powerful-frontier-ai-models/">violation</a>. (see references)</p><p>New York’s amended RAISE Act adopted California’s 10²⁶ threshold. One source says a revenue test was expected to replace the original $100 million compute-cost test, so check the enacted text.</p><p>So here’s the layman’s version. Frontier AI is the best AI available at this moment, measured against today’s standards, and the answer changes as the field moves.</p><p>But of course, that raises a bigger question. If the line keeps moving, who gets to draw it? Governments, companies, and researchers all have different answers, and they don’t agree. That’s the next <a href="https://santhilatakv.medium.com/why-frontier-depends-on-the-law-612a7b40a0be?postPublishedType=initial">post</a>.</p><p><em>References:</em></p><ul><li>CASRAI, “What is a frontier AI model?” <a href="https://casrai.org/guides/what-is-a-frontier-ai-model">https://casrai.org/guides/what-is-a-frontier-ai-model</a></li><li>Cato Networks, “Frontier AI” explainer. <a href="https://www.catonetworks.com/?p=168150">https://www.catonetworks.com/?p=168150</a></li><li>CSET, Georgetown University, on defining frontier AI for regulation. <a href="https://cset.georgetown.edu/?p=16031">https://cset.georgetown.edu/?p=16031</a></li><li>ITIF / Data Innovation, “AI’s Frontier Is Moving. Its Legal Definition Should Too” (Aug 2026). <a href="https://itif.org/publications/2026/08/28/ais-frontier-is-moving-its-legal-definition-should-too/">https://itif.org/publications/2026/08/28/ais-frontier-is-moving-its-legal-definition-should-too/</a></li><li>EU AI Act GPAI guidelines overview: <a href="https://artificialintelligenceact.eu/gpai-guidelines-overview/">https://artificialintelligenceact.eu/gpai-guidelines-overview/</a></li><li>Orrick, on California’s TFAIA (SB 53): <a href="https://www.orrick.com/en/Insights/2025/10/Californias-Transparency-in-Frontier-Artificial-Intelligence-Act-5-Things-You-Should-Know">https://www.orrick.com/en/Insights/2025/10/Californias-Transparency-in-Frontier-Artificial-Intelligence-Act-5-Things-You-Should-Know</a></li><li>Future of Privacy Forum, “California’s SB 53: The First Frontier AI Law, Explained”: <a href="https://fpf.org/blog/californias-sb-53-the-first-frontier-ai-law-explained/">https://fpf.org/blog/californias-sb-53-the-first-frontier-ai-law-explained/</a></li><li>Cooley, “New York’s Frontier AI Law Gets a California Makeover”: <a href="https://www.cooley.com/news/insight/2026/2026-03-31-new-yorks-frontier-ai-law-gets-a-california-makeover-with-some-key-differences">https://www.cooley.com/news/insight/2026/2026-03-31-new-yorks-frontier-ai-law-gets-a-california-makeover-with-some-key-differences</a></li><li>Nat Law Review, on the original RAISE Act: <a href="https://natlawreview.com/article/new-york-passes-responsible-ai-safety-and-education-act">https://natlawreview.com/article/new-york-passes-responsible-ai-safety-and-education-act</a></li><li>Reason Foundation, “California’s AI law works by staying narrow”: <a href="https://reason.org/commentary/californias-ai-law-works-by-staying-narrow/">https://reason.org/commentary/californias-ai-law-works-by-staying-narrow/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3098682a5a5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Agentic Newsroom — How autonomous services turn headlines into answers — Post 1]]></title>
            <link>https://santhilatakv.medium.com/the-agentic-newsroom-how-autonomous-services-turn-headlines-into-answers-post-1-967306121b02?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/967306121b02</guid>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Sat, 25 Apr 2026 11:36:04 GMT</pubDate>
            <atom:updated>2026-04-25T11:36:04.291Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Bootstrapping an Agentic Media-Intelligence Stack (Infra + Repo Skeleton)</blockquote><p><strong>A Brief Recap</strong></p><p>For the detailed introduction about the <strong>Multi-Agent Media Intelligence (MAMI)</strong> project, please visit previous posts in this series:</p><ul><li><a href="https://santhilatakv.medium.com/the-agentic-newsroom-how-autonomous-services-turn-headlines-into-answers-post-0-916e76d1ae85">Post 0</a> — Describes what and why of the architectural decisions</li><li>Post 1 (This post) — Bootstrapping an Agentic Media-Intelligence Stack (Infra + Repo Skeleton)</li><li><a href="https://github.com/KuppiliVenkataS/MultiAgent_MediaIntelligence">GitHub</a> — This is a work in progress.</li></ul><p>The MAMI is an event-driven platform where specialist AI agents continuously</p><ul><li>Ingest articles, posts, podcasts, and videos,</li><li>Normalize and enrich the entities, events, claims, and sentiment,</li><li>Reason and decide on the relevance of a topic, and what’s trending, and importantly, what’s the stance of this article?,</li><li>Index and link knowledge (Knowledge Graph + search),</li><li>Answer and act for users about news briefs, alerts, and Q&amp;A with citations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k8NJ7GGhaWPMPBKuNnIALA.jpeg" /><figcaption>The basic workflow of the MAMI process (image generated by AI)</figcaption></figure><p><strong>The building blocks of MAMI are,</strong></p><ul><li><strong>Ingest</strong> — raw documents of articles, posts, podcasts, and videos from various sources</li><li><strong>Normalize</strong> <strong>and enrich</strong> — entities, events, claims, and get the sentiment,</li><li><strong>Reason and decide</strong> on the relevance of a topic, and what’s trending, and importantly, what’s the stance of this article?,</li><li><strong>Index and link</strong> — knowledge (Knowledge Graph + search),</li><li><strong>Answer and act </strong>for users about briefs, alerts, and Q&amp;A with citations.</li></ul><h3><strong>Bootstrapping an Agentic Media-Intelligence Stack (Infra + Repo Skeleton)</strong></h3><p>Only important folders of the MultiAgent_MediaIntelligence repo (GitHub link above) useful for this post are shown below. Let’s spin up infra (Kafka, Redis, Neo4j, Elasticsearch, Kibana) and a FastAPI service.</p><p><strong>Why do we need this complex structure?</strong></p><p>Media intelligence needs continuous crawling, enrichment, and reasoning from the continuously published events. A plain web app is not sufficient. We need an event-driven backbone (like Kafka), stateful stores to store the knowledge (e.g, Neo4j, Elasticsearch), and services that we can develop using a multi-agent system.</p><p>I would like to concentrate on addressing the common Docker/Compose challenges I encountered as a beginner, aiming to establish a clean and efficient development loop.</p><p>├─ apps/<br>│ ├─ api/ # FastAPI service<br>│ ├─ orchestrator/ # (future) task router/agent coordinator<br>│ └─ ingest/ <br>├─ infra/<br>│ └─ neo4j/ # neo4j.conf <br>├─ scripts/<br>│ └─ create_topics.sh # Kafka topics helper<br>├─ tests/ # pytest tests<br>├─ docker-compose.yml<br>├─ Makefile<br>├─ .env # runtime config (Compose reads this) — shown as .env.example in repo<br>└─ requirements.txt</p><blockquote>pitfall 1: env/ or .venv/ is your Python virtual environment. Please rename if need be. Because we have a .env file, a key–value config (Compose auto-loads it). The .env file is different from your virtual environment folder env/. You often need both.</blockquote><ol><li>Neo4j for entities/events/relationships (suitable for graph provision)</li></ol><p><strong>docker-compose.yml</strong></p><p>Docker Compose is a tool used to define and run multi-container Docker applications and services. Instead of using multiple individual commands to run each container, Compose allows you to use a single configuration file to manage all of your application’s services, networks, and volumes. Check the <a href="https://github.com/KuppiliVenkataS/MultiAgent_MediaIntelligence/blob/main/docker-compose.yml">file</a>.</p><p>A quick reminder: install Docker on the system. It is not necessary to have Docker if you intend to develop and run from your desktop. But nice to have a Docker set up to freeze the versions that run successfully.</p><p><strong>Key services &amp; why:</strong></p><ul><li><strong>Kafka + Zookeeper</strong> — event backbone(1,2)</li><li><strong>Redis</strong> — fast cache/queues (dedupe, rate limits)</li><li><strong>Neo4j</strong> — knowledge graph</li><li><strong>Elasticsearch + Kibana</strong> — search + quick viz</li></ul><p><strong>apps/ingest/models.py</strong></p><p><strong>The Event-driven, multi-agent pipeline with a Kafka backbone</strong></p><p>An event-driven, multi-agent pipeline with a Kafka backbone is an architectural pattern for building highly scalable and resilient data processing systems (1). Apache Kafka is a distributed event streaming platform that serves as the central nervous system of the pipeline (2). Instead of tasks being initiated by a central scheduler, the Event-Driven Architecture allows initiation by the arrival of a new event in a Kafka topic (highly reactive to real-time data). The multi-agent pipeline refers to the series of specialized “agents” or microservices that perform specific tasks.</p><p>Our architecture is supported by two read models:</p><ol><li>Elasticsearch for fast full-text/search and,</li><li>Neo4j for entities/events/relationships (suitable for graph provision)</li></ol><p><strong>References (and links):</strong></p><p><strong>References (and links):</strong></p><ol><li><a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#:~:text=ZooKeeper%20stores%20its%20application%20configuration,in%20the%20zk%2D0%20Pod">https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#:~:text=ZooKeeper%20stores%20its%20application%20configuration,in%20the%20zk%2D0%20Pod</a>.</li><li><a href="https://medium.com/@turkishtechnology/a-deep-dive-into-distributed-messaging-system-kafka-39304155377c">https://medium.com/@turkishtechnology/a-deep-dive-into-distributed-messaging-system-kafka-39304155377c</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=967306121b02" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Strait of Hormuz and Trillions of Defense Bills]]></title>
            <link>https://santhilatakv.medium.com/beyond-the-strait-of-hormuz-and-trillions-of-defense-bills-194404333f26?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/194404333f26</guid>
            <category><![CDATA[global-stability]]></category>
            <category><![CDATA[future-generations]]></category>
            <category><![CDATA[peace-economy]]></category>
            <category><![CDATA[youth-first]]></category>
            <category><![CDATA[mental-health-matters]]></category>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Mon, 06 Apr 2026 07:23:46 GMT</pubDate>
            <atom:updated>2026-04-06T07:23:46.233Z</atom:updated>
            <content:encoded><![CDATA[<h4>… lies an invisible debt — the unprocessed grief of a global generation</h4><p>While global headlines focus on the <strong>20% disruption of global oil supplies,</strong> a more dangerous deficit is quietly accumulating. It is the diversion of our most precious resource: <strong>The potential of the next generation. </strong>Every hour the global economy remains on a “war footing” is an hour in which thousands of schools remain unbuilt and medical breakthroughs remain unfunded. This is a <strong>civilizational trade-off</strong> that no country, regardless of its wealth, can afford indefinitely.</p><p>This is the second part of the effect of the horrors of war on future generations and the repercussions. Read the context for this post <a href="https://medium.com/@santhilatakv/the-trauma-nobodys-accounting-for-8dae4f087849">here</a>.</p><p><strong>The Absurdity of the “Balance”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tOmfg71r3eeoAhaW6sBIBA.png" /><figcaption>An AI-generated image depicting a massive misallocation of global resources</figcaption></figure><p>The graphic above uses a metaphorical scale where the $2.6 trillion of military spending is <em>actually</em> what the world is prioritizing, while the massive, <strong>unfulfilled need</strong> on the other side is what we claim to value. This infographic is only a representation to give a sense of profound imbalance and visual hypocrisy. Instead of a sense of helplessness, we must <strong>shift resources</strong> from the military column to the sustainable development column to achieve <strong>Generational Stability</strong>.</p><p><strong>Global Secondary Trauma</strong></p><p>Here is the heat map to visually prove that modern conflict is a global health epidemic. It is an attempt to convey a serious and urgent message using a data-analytical visual. This represents the physical destruction may be at the epicentre of the war zone. But the trauma and repercussions can spread over the far away parts of the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HG_tk2SEI3IXH0hlzImk3Q.png" /><figcaption>AI-generated image visualizing Global Secondary Trauma</figcaption></figure><p>The primary target zone shown in red concentrated, jagged rings that are geolocated precisely at the <strong>active conflict zones</strong> (the epicenter). This is the traditional, limited view of war, where the bombs are falling.</p><p>The branching, fiber-optic-like white thin lines represent the <strong>Digital Vectors</strong> (social media feeds and 24-hour news cycles). They act as the delivery system, directly connecting the physical violence in the epicenter to population centers thousands of miles away.</p><p><strong>The Global Anxiety Spikes </strong>in orange and deep purple<strong> </strong>is the “Secondary Trauma” heat map. It represents an aggregated <strong>global youth trauma index</strong>, drawing from anonymized digital signals like search trends (for example, “nuclear war,” “draft”), social media sentiment analysis (distress signals), and crisis hotline spikes. The massive, pulsating <strong>Trauma Spikes</strong> in orange/purple blobs are ablaze in non-combatant, globally connected urban hubs. The anxiety spreads across the world. This visual is designed to shatter the illusion of distance, proving that a conflict in one region is an invasion of the mental space of youth everywhere.</p><p><strong>The Hidden Debt — the real cost of conflict</strong></p><p>The cost of conflict needs to be measured in generations, not in barrels of oil, as the <strong>economic ghost</strong> <strong>of unprocessed grief</strong> is a productivity loss that can last for 50 years after a war ends.</p><p>The most dangerous weapon of modern war is the unprocessed grief we hand over to the next generation. It is time to stop financing the shadow and start investing in the light. 💙</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=194404333f26" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI First Vs Data First — Neither]]></title>
            <link>https://santhilatakv.medium.com/ai-first-vs-data-first-neither-95eb95011ffe?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/95eb95011ffe</guid>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[incremental-development]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 19:07:05 GMT</pubDate>
            <atom:updated>2026-04-01T19:22:22.797Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="AI_first or Data First — Neither" src="https://cdn-images-1.medium.com/max/1024/1*cXtqxWs-MrCi552dSu9lxg.png" /><figcaption>AI-generated Image Depicting what really matters: AI First or Data First</figcaption></figure><p>The debate around <em>“AI-first vs Data-first”</em> is fundamentally flawed. It frames the problem as a sequencing decision, when in reality it is a systems design problem. AI-first without discipline creates brittle, non-deterministic solutions. Data-first initiatives, especially in large enterprises, often become multi-year transformation programs with delayed value. In regulated industries like BFSI, both approaches introduce risk, either through uncontrolled AI or through strategic stagnation.</p><p>The real issue is not whether data comes before AI, but whether the data is <strong>governed when AI begins to use it</strong>. Poorly governed data leads directly to hallucinations, bias, inconsistent outputs, and regulatory exposure. Lineage, quality controls, access governance, and auditability are not optional; they are structural requirements. Without them, data is not an asset; it is a hidden liability embedded inside every AI decision.</p><p>However, it is equally important to recognise a critical gap in how organisations think about governance: <strong>data governance is not the same as AI governance</strong>. Data governance ensures that inputs are controlled, but AI systems introduce new dimensions. Such as model behaviour, decision impact, non-determinism, and real-world consequences. You can have well-governed data and still produce biased, unsafe, or unreliable AI outcomes. True AI governance must extend beyond data to include model controls, decision validation, and continuous monitoring.</p><p>At the same time, waiting for “perfect data” is a fallacy. Data maturity does not emerge in isolation. It is shaped through usage. <strong>AI systems, when introduced early, act as a forcing function on data quality.</strong> They expose gaps in completeness, inconsistencies in definitions, and breakdowns in pipelines far more effectively than static data programs. In this sense, AI is not just a consumer of data. It is also a <strong>testing mechanism for the data itself</strong>.</p><p>Without AI-driven validation, data platforms often drift. Completeness is assumed rather than tested, definitions diverge across teams, and governance frameworks become documentation-heavy but insight-light. Critically, governance without AI feedback loops may fail to capture real-world bias or decision impact. <strong>You end up governing data in theory, not in practice.</strong> AI introduces the necessary pressure to continuously validate whether both the data and the decisions it drives are fit for purpose.</p><p>Consider these examples: Across advisory, consulting, and financial services engagements, a consistent pattern emerges -</p><ul><li>Revenue leakage programs with “clean data” still fail until AI exposes behavioural inconsistencies</li><li>Fraud systems remain reactive until AI forces continuous data evolution</li></ul><p><strong>In each case, AI use cases did not just use data; they tested it. </strong>And in doing so, it revealed a critical truth: Data governance alone cannot guarantee AI success. AI governance must validate decisions, not just inputs.</p><p>The winning approach, therefore, is neither AI-first nor Data-first. It is <strong>Governance-first, Incremental AI</strong>. Start with focused use cases. Apply guardrails from day one, validation layers, human-in-the-loop, monitoring, and audit trails. Let AI usage continuously refine data quality, governance, and model performance. Build the system iteratively, but with structural integrity. That is how organisations move from experimental AI to audit-ready, enterprise-grade capability without waiting indefinitely or failing prematurely.</p><p><strong>If your data has never been tested by AI, you don’t know if it’s fit for purpose.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95eb95011ffe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Trauma Nobody’s Accounting For]]></title>
            <link>https://santhilatakv.medium.com/the-trauma-nobodys-accounting-for-8dae4f087849?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/8dae4f087849</guid>
            <category><![CDATA[war]]></category>
            <category><![CDATA[youth-first]]></category>
            <category><![CDATA[save-our-children]]></category>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 07:47:52 GMT</pubDate>
            <atom:updated>2026-03-16T07:47:52.451Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ynTIdPhvKCzooVYNVKeO9w.png" /></figure><p>While the world debates military costs and geopolitical shifts, <br>No one is measuring the intergenerational psychological trauma passed from parents to children, from the conflict zones, even long after the last missile fired!</p><p>News channels and social media are obsessed over displaced populations and disrupted supply chains, yet ignore the rerouting of young people’s sense of safety and belonging. This is a crisis with no satellite imagery and no official briefing.</p><p>As global citizens, we must demand that peace frameworks center on mental health reconstruction and youth-First diplomacy, because the most dangerous weapon of modern war is the “unprocessed grief” we hand over to the next generation. 💙</p><p>Save our children from war horrors!</p><p><a href="https://www.linkedin.com/search/results/all/?keywords=%23saveourchildren&amp;origin=HASH_TAG_FROM_FEED">#SaveOurChildren</a> <a href="https://www.linkedin.com/search/results/all/?keywords=%23futuregenerations&amp;origin=HASH_TAG_FROM_FEED">#FutureGenerations</a> <a href="https://www.linkedin.com/search/results/all/?keywords=%23mentalhealth&amp;origin=HASH_TAG_FROM_FEED">#MentalHealth</a> <a href="https://www.linkedin.com/search/results/all/?keywords=%23nomorewar&amp;origin=HASH_TAG_FROM_FEED">#NoMoreWar</a> <a href="https://www.linkedin.com/search/results/all/?keywords=%23youthfirst&amp;origin=HASH_TAG_FROM_FEED">#YouthFirst</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8dae4f087849" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Agentic Newsroom — How autonomous services turn headlines into answers — Post 2]]></title>
            <link>https://santhilatakv.medium.com/the-agentic-newsroom-how-autonomous-services-turn-headlines-into-answers-post-2-6a1a6c5b9dc8?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a1a6c5b9dc8</guid>
            <category><![CDATA[enrichment]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[media-intelligence]]></category>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Mon, 08 Sep 2025 09:12:34 GMT</pubDate>
            <atom:updated>2025-09-08T16:41:20.442Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Agentic Newsroom — How autonomous services turn headlines into answers — Post 2</h3><blockquote>Stream, Consume, Enrich (Kafka in the Loop)</blockquote><p>The MAMI is an event-driven platform where specialist AI agents continuously ingest articles, posts, podcasts, and videos, <strong>normalize and enrich the entities, events, claims, and sentiment. </strong>Reason and decide on the relevance of a topic, and what’s trending, and importantly, what’s the stance of this article? index &amp; link knowledge (Knowledge Graph + search), and answer and act for users about news briefs, alerts, and Q&amp;A with citations.</p><p><strong>Are we there yet in declaring agentic?</strong></p><p>The answer is a resounding YES. With RSS feeds in (previous posts), entities emerge, and we’re officially “A<strong>gentic</strong>”! 🎉🥳</p><p><strong>A brief recap:</strong></p><p>For the detailed introduction about the <strong>Multi-Agent Media Intelligence (MAMI)</strong> project, please visit previous posts in this series:</p><ul><li><a href="https://santhilatakv.medium.com/the-agentic-newsroom-how-autonomous-services-turn-headlines-into-answers-post-0-916e76d1ae85">Post 0</a> — Describes what and why of the architectural decisions</li><li>P<a href="https://santhilatakv.medium.com/the-agentic-newsroom-how-autonomous-services-turn-headlines-into-answers-post-1-3305e2b40eba">ost 1</a> — Bootstrapping an Agentic Media-Intelligence Stack (Infra + Repo Skeleton)</li><li>Post 2 (This post)— Stream, Consume, Enrich (Kafka in the Loop)</li><li><a href="https://github.com/KuppiliVenkataS/MultiAgent_MediaIntelligence">GitHub</a> — This is a work in progress.</li></ul><p><strong>Stream, Consume, Enrich (Kafka in the Loop)</strong></p><p>In this post, we discuss the functionality, operational details of normalising and enriching messages and possible pitfalls in implementing. We bring Kafka online, stream normalized documents from the ingester, consume them in the <em>Enricher</em>, and emit enriched records (NER, sentiment, stance, basic claims). We’ll see how end-to-end data is moving and learn quick debug moves.</p><p><strong>Prereqs</strong></p><ul><li>Check <em>colima</em> running. (Colima is an open-source tool that lets us run Docker and containerd in a virtual machine (VM) on macOS and Linux)</li><li>Start services.</li></ul><pre>colima status <br># if not running, start it with the following bash command<br><br># no of virtual cpus-4, allocates 6GB of RAM, sets the size of the VM&#39;s disk to 60 GB<br>colima start --cpu 4 --memory 6 --disk 60 <br><br># check whether the docker context is colima<br>docker context use colima<br>docker info<br><br># check status of docker services. None of them are up yet.<br># Notice, I am using docker-compose on my MacOS. You may need docker compose (no hyphen for V2)<br>docker-compose ps<br><br>#start services <br>docker-compose up -d zookeeper kafka redis<br><br># if you want check the status of these services<br>docker-compose ps<br><br>#we have topics (raw, enridhed and error documents) created earlier. check<br>bash scripts/create_topics.sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNUCfwpNUhxX70tjc7ujTw.png" /><figcaption>Depiction of the enricher module generated using AI</figcaption></figure><p><strong>The <em>enricher </em>module</strong></p><pre># <br>apps/<br>  api/<br>  orchestrator/<br>  ingest/<br>  enricher/<br>  ...<br>...</pre><p>News ‘Enrichment’ is where raw headlines become structured signals our system can actually reason over. For every normalized document, we <em>extract</em> entities (people, orgs, places, etc) with <em>spaCy</em>, score <em>sentiment</em> with a fast polarity model (VADER), infer <em>stance</em> toward key entities via lightweight lexical cues, and mine <em>claims</em> using high-precision patterns (“plans to…”, “announced…”, “up 10% YoY”).</p><p>Since this is a demo version, each step is cost-effective (We only run on English text, cap input length, and keep models cheap by default so the stream stays fast). The result is an <em>enriched-document</em> that carries both the original context and machine-readable fields such as entities[], sentiment (label and compound), stance[], and claims[]</p><pre>unlock filtering (&quot;positive coverage of ORG X&quot;)<br>routing (send M&amp;A claims to finance)</pre><p>The schema is versioned and additive; we can A/B new extractors, roll forward safely, and keep downstream consumers stable when the agents get smarter (later versions).</p><p><strong>Explanation of the code blocks in the enricher module</strong></p><ul><li><em>apps/enricher/models.py — </em>Pydantic models for the enricher’s input/output contract. We keep the schema additive and JSON-friendly so downstream consumers (indexers, API, analytics) can evolve independently. These models intentionally mirror a subset of the ingest document plus new enrichment fields.</li><li><em>apps/enricher/nlp.py</em> — Entity extraction helpers. We load spaCy once per process, cap input length to keep latency predictable, and filter to a curated label set to reduce noise.</li><li><em>apps/enricher/sentiment.py</em> — Document-level sentiment using VADER (this is a rule-based, fast, and works well on headlines/business news). We keep a simple mapping to ‘neg/neu/pos’ for filtering and store the raw compound score.</li><li><em>apps/enricher/stance.py</em> —Stance estimation toward entities. By default, lexical rule-based uses local context windows around the entity span.</li><li><em>apps/enricher/claims.py</em> —Basic claim mining via high-precision regex patterns on sentence segments. This is a pragmatic baseline: it catches a large share of business-news actions (e.g., ‘announced’, ‘plans to’, ‘up 10%’) at very low cost and with clear cues.</li><li><em>apps/enricher/worker.py<br>- </em>Kafka consumer → enrichment → producer worker. <br>- Consumes normalized documents from RAW_TOPIC. Runs NER, sentiment, stance, and claim extraction (English-gated). Produces EnrichedDocument to ENRICHED_TOPICS.<br>- On exceptions, emits a context payload to ERROR_TOPICS and skips committing that offset so the message can be retried.</li></ul><p>How to run?</p><pre>docker-compose build enricher<br>docker-compose up -d --force-recreate enricher<br>docker-compose logs -f --tail=120 enricher</pre><p><strong>How does the success of the <em>enricher</em> module look?</strong></p><pre>docker-compose exec -T kafka kafka-console-consumer \<br>  --bootstrap-server kafka:9092 --topic raw-documents --from-beginning --max-messages 1<br><br>{<br>&quot;id&quot;: &quot;3fc7e9bc0376c14a909cf2ea923ae43c90fd8790&quot;, <br>&quot;source_url&quot;: &quot;https://www.theverge.com/news/773567/vodafone-generative-ai-ad-presenter-tiktok&quot;, <br>&quot;source_type&quot;: &quot;rss&quot;, <br>&quot;title&quot;: &quot;The influencer in this AI Vodafone ad isn’t real&quot;, <br>&quot;text&quot;: &quot;I opened TikTok while visiting Germany last week and stumbled across<br> a Vodafone ad being presented by a woman who probably doesn’t exist. <br>The ad includes several “tells” that suggest the presenter was artificially <br>created using generative AI — the unnatural style and movement of her hair, <br>the way her moles disappear, and the uncanny valley vibe of her facial <br>expressions — and Vodafone confirmed my suspicions in its video comments.<br>The influencer in this AI Vodafone ad isn’t real<br>Vodafone is testing the artificial presenter because ‘AI is so much a<br> part of everyday life.’<br>Vodafone is testing the artificial presenter because ‘<br>AI is so much a part of everyday life.’<br>In response to a commenter asking why Vodafone couldn’t put <br>“a real person in front of the camera,” the company said it was<br> “testing different styles of advertising — this time with AI,” <br>according to machine translation of the German text. <br>In another comment, Vodafone said that <br>“AI is so much a part of everyday life these days that we also try<br> it out in advertising.”\nAs an ad, it certainly captured my attention,<br> but only because my brain registered that there was something “off.”<br>This isn’t the first time that Vodafone has embraced generative AI in <br>its ads, having released a commercial last year in which every shot was <br>AI-generated. Fake influencers created using artificial intelligence <br>are also rising in popularity, according to a New York Times report, <br>with notable examples like Lil’ Miquela — a creation of tech company<br> Dapper Labs — having already appeared in campaigns for Calvin Klein, <br>Prada, and BMW.&quot;, <br>&quot;lang&quot;: &quot;en&quot;, <br>&quot;published_at&quot;: null, <br>&quot;authors&quot;: [], <br>&quot;media&quot;: [], <br>&quot;raw&quot;: {}<br>}</pre><pre>docker-compose exec -T kafka kafka-console-consumer \<br>  --bootstrap-server kafka:9092 --topic enriched-documents --from-beginning --max-messages 1<br>{<br>&quot;id&quot;: &quot;3fc7e9bc0376c14a909cf2ea923ae43c90fd8790&quot;, <br>&quot;source_url&quot;: &quot;https://www.theverge.com/news/773567/vodafone-generative-ai-ad-presenter-tiktok&quot;, <br>&quot;source_type&quot;: &quot;rss&quot;, <br>&quot;title&quot;: &quot;The influencer in this AI Vodafone ad isn’t real&quot;, <br>&quot;lang&quot;: &quot;en&quot;, <br>&quot;entities&quot;: [{&quot;text&quot;: &quot;TikTok&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 9, &quot;end&quot;: 15}, <br>             {&quot;text&quot;: &quot;Germany&quot;, &quot;label&quot;: &quot;GPE&quot;, &quot;start&quot;: 31, &quot;end&quot;: 38}, <br>             {&quot;text&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 71, &quot;end&quot;: 79}, <br>             {&quot;text&quot;: &quot;AI&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 239, &quot;end&quot;: 241}, <br>             {&quot;text&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 379, &quot;end&quot;: 387}, <br>             {&quot;text&quot;: &quot;AI Vodafone&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 458, &quot;end&quot;: 469},<br>             {&quot;text&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 484, &quot;end&quot;: 492}, <br>             {&quot;text&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 578, &quot;end&quot;: 586}, <br>             {&quot;text&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 710, &quot;end&quot;: 718}, <br>             {&quot;text&quot;: &quot;AI&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 854, &quot;end&quot;: 856}, <br>             {&quot;text&quot;: &quot;German&quot;, &quot;label&quot;: &quot;NORP&quot;, &quot;start&quot;: 899, &quot;end&quot;: 905}, <br>             {&quot;text&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 932, &quot;end&quot;: 940}, <br>              {&quot;text&quot;: &quot;AI&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 952, &quot;end&quot;: 954}, <br>              {&quot;text&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 1188, &quot;end&quot;: 1196}, <br>              {&quot;text&quot;: &quot;AI&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 1221, &quot;end&quot;: 1223}, <br>              {&quot;text&quot;: &quot;New York Times&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 1414, &quot;end&quot;: 1428}, <br>              {&quot;text&quot;: &quot;Lil’ Miquela&quot;, &quot;label&quot;: &quot;PERSON&quot;, &quot;start&quot;: 1464, &quot;end&quot;: 1476}, <br>              {&quot;text&quot;: &quot;Dapper Labs&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 1506, &quot;end&quot;: 1517}, <br>              {&quot;text&quot;: &quot;Calvin Klein&quot;, &quot;label&quot;: &quot;PERSON&quot;, &quot;start&quot;: 1561, &quot;end&quot;: 1573}, <br>              {&quot;text&quot;: &quot;Prada&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 1575, &quot;end&quot;: 1580}, <br>              {&quot;text&quot;: &quot;BMW&quot;, &quot;label&quot;: &quot;ORG&quot;, &quot;start&quot;: 1586, &quot;end&quot;: 1589}], <br>&quot;extra&quot;: {}, <br>&quot;sentiment&quot;: {&quot;label&quot;: &quot;pos&quot;, &quot;compound&quot;: 0.9109}, <br>&quot;stance&quot;: [{&quot;target&quot;: &quot;TikTok&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Germany&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;AI&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;AI Vodafone&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;AI&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;German&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;AI&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Vodafone&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;AI&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;New York Times&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Lil’ Miquela&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Dapper Labs&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Calvin Klein&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;Prada&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}, <br>            {&quot;target&quot;: &quot;BMW&quot;, &quot;label&quot;: &quot;unknown&quot;, &quot;confidence&quot;: 0.0}], <br>&quot;claims&quot;: [{&quot;text&quot;: &quot;In response to a commenter asking why Vodafone <br>                     couldn’t put “a real person in front of the camera,” the company said it was <br>                    “testing different styles of advertising — this time with AI,” <br>                    according to machine translation of the German text.&quot;, <br>                    &quot;kind&quot;: &quot;uncertain&quot;, <br>                    &quot;cues&quot;: [&quot;\\b(report[s]? that|according to)\\b&quot;], <br>                    &quot;confidence&quot;: 0.6}, <br>           {&quot;text&quot;: &quot;Fake influencers created using artificial intelligence are also <br>                     rising in popularity, according to a New York Times report, with notable <br>                     examples like Lil’ Miquela — a creation of tech company Dapper Labs — having <br>                     already appeared in campaigns for Calvin Klein, Prada, and BMW.&quot;, <br>                   &quot;kind&quot;: &quot;uncertain&quot;,  <br>                   &quot;cues&quot;: [&quot;\\b(report[s]? that|according to)\\b&quot;], <br>                   &quot;confidence&quot;: 0.6}]}</pre><p><strong>You will see that the output meets the contract</strong></p><pre>{<br>  &quot;id&quot;: &quot;sha1-or-stable-id&quot;,<br>  &quot;source_url&quot;: &quot;https://…&quot;,<br>  &quot;source_type&quot;: &quot;rss&quot;,<br>  &quot;title&quot;: &quot;…&quot;,<br>  &quot;lang&quot;: &quot;en&quot;,<br>  &quot;entities&quot;: [{&quot;text&quot;:&quot;Apple&quot;,&quot;label&quot;:&quot;ORG&quot;,&quot;start&quot;:0,&quot;end&quot;:5}],<br>  &quot;sentiment&quot;: {&quot;label&quot;:&quot;pos&quot;,&quot;compound&quot;: 0.6},<br>  &quot;stance&quot;: [{&quot;target&quot;:&quot;Apple&quot;,&quot;label&quot;:&quot;pro&quot;,&quot;confidence&quot;:0.7}],<br>  &quot;claims&quot;: [{&quot;text&quot;:&quot;Apple plans to…&quot;,&quot;kind&quot;:&quot;commitment&quot;,&quot;cues&quot;:[&quot;\\b(plans to|aims to|intends to)\\b&quot;],&quot;confidence&quot;:0.6}],<br>  &quot;extra&quot;: {}<br>}</pre><p><strong>Latency &amp; throughput are reasonable</strong></p><ul><li>Per-message enrich latency (ingest → enriched): typically &lt; 300 ms on small model settings.</li><li>Throughput: keep up with your input rate (near-zero consumer lag at steady state).</li></ul><pre>docker-compose logs --tail=200 enricher | tail -n +1<br># Look for log lines like:<br># [enricher] &lt;id&gt; ents=7 stance=3 claims=2 sent=pos -&gt; enriched-documents:1</pre><p><strong>Check error-documents</strong></p><pre>docker-compose exec -T kafka kafka-console-consumer \<br>  --bootstrap-server kafka:9092 --topic errors-documents --from-beginning --timeout-ms 3000</pre><p>So, this is all for this post. Until now, we have consumed the raw messages and enriched them. The next post in this series will look into Elastic Search.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a1a6c5b9dc8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Agentic Newsroom — How autonomous services turn headlines into answers — Post 1]]></title>
            <link>https://santhilatakv.medium.com/the-agentic-newsroom-how-autonomous-services-turn-headlines-into-answers-post-1-3305e2b40eba?source=rss-444748761ef8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3305e2b40eba</guid>
            <category><![CDATA[rss-feeds]]></category>
            <category><![CDATA[kafka-producer]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[media-intelligence]]></category>
            <dc:creator><![CDATA[Santhilata KV]]></dc:creator>
            <pubDate>Fri, 05 Sep 2025 10:15:49 GMT</pubDate>
            <atom:updated>2025-09-08T09:13:22.170Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Agentic Newsroom — How autonomous services turn headlines into answers — Post 1</h3><blockquote>Bootstrapping an Agentic Media-Intelligence Stack (Infra + Repo Skeleton)</blockquote><p><strong>A Brief Recap</strong></p><p>For the detailed introduction about the <strong>Multi-Agent Media Intelligence (MAMI)</strong> project, please visit previous posts in this series:</p><ul><li><a href="https://santhilatakv.medium.com/the-agentic-newsroom-how-autonomous-services-turn-headlines-into-answers-post-0-916e76d1ae85">Post 0</a> — Describes what and why of the architectural decisions</li><li>Post 1 (This post) — Bootstrapping an Agentic Media-Intelligence Stack (Infra + Repo Skeleton)</li><li><a href="https://santhilatakv.medium.com/the-agentic-newsroom-how-autonomous-services-turn-headlines-into-answers-post-2-6a1a6c5b9dc8">Post 2</a> — Stream, Consume, Enrich (Kafka in the Loop)</li><li><a href="https://github.com/KuppiliVenkataS/MultiAgent_MediaIntelligence">GitHub</a> — This is a work in progress.</li></ul><p>The MAMI is an event-driven platform where specialist AI agents continuously</p><ul><li>Ingest articles, posts, podcasts, and videos,</li><li>Normalize and enrich the entities, events, claims, and sentiment,</li><li>Reason and decide on the relevance of a topic, and what’s trending, and importantly, what’s the stance of this article?,</li><li>Index and link knowledge (Knowledge Graph + search),</li><li>Answer and act for users about news briefs, alerts, and Q&amp;A with citations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k8NJ7GGhaWPMPBKuNnIALA.jpeg" /><figcaption>The basic workflow of the MAMI process (image generated by AI)</figcaption></figure><p><strong>The building blocks of MAMI are,</strong></p><ul><li><strong>Ingest</strong> — raw documents of articles, posts, podcasts, and videos from various sources</li><li><strong>Normalize</strong> <strong>and enrich</strong> — entities, events, claims, and get the sentiment,</li><li><strong>Reason and decide</strong> on the relevance of a topic, and what’s trending, and importantly, what’s the stance of this article?,</li><li><strong>Index and link</strong> — knowledge (Knowledge Graph + search),</li><li><strong>Answer and act </strong>for users about briefs, alerts, and Q&amp;A with citations.</li></ul><h3>Bootstrapping an Agentic Media-Intelligence Stack (Infra + Repo Skeleton)</h3><p>Only important folders of the MultiAgent_MediaIntelligence repo (GitHub link above) useful for this post are shown below. Let’s spin up infra (Kafka, Redis, Neo4j, Elasticsearch, Kibana) and a FastAPI service.</p><p><strong>Why do we need this complex structure?</strong></p><p>Media intelligence needs continuous crawling, enrichment, and reasoning from the continuously published events. A plain web app is not sufficient. We need an event-driven backbone (like Kafka), stateful stores to store the knowledge (e.g, Neo4j, Elasticsearch), and services that we can develop using a multi-agent system.</p><p>I would like to concentrate on addressing the common Docker/Compose challenges I encountered as a beginner, aiming to establish a clean and efficient development loop.</p><p>├─ apps/<br>│ ├─ api/ # FastAPI service<br>│ ├─ orchestrator/ # (future) task router/agent coordinator<br>│ └─ ingest/ <br>├─ infra/<br>│ └─ neo4j/ # neo4j.conf <br>├─ scripts/<br>│ └─ create_topics.sh # Kafka topics helper<br>├─ tests/ # pytest tests<br>├─ docker-compose.yml<br>├─ Makefile<br>├─ .env # runtime config (Compose reads this) — shown as .env.example in repo<br>└─ requirements.txt</p><blockquote>pitfall 1: <em>env/</em> or <em>.venv/</em> is your Python virtual environment. Please rename if need be. Because we have a .env file, a key–value config (Compose auto-loads it). The .env file is different from your virtual environment folder env/. You often need both.</blockquote><p><strong>docker-compose.yml</strong></p><p>A quick reminder: install Docker on the system. It is not necessary to have Docker if you intend to develop and run from your desktop. But nice to have a Docker set up to freeze the versions that run successfully.</p><p>Docker Compose is a tool used to define and run multi-container Docker applications and services. Instead of using multiple individual commands to run each container, Compose allows you to use a single configuration file to manage all of your application’s services, networks, and volumes. Check the <a href="https://github.com/KuppiliVenkataS/MultiAgent_MediaIntelligence/blob/main/docker-compose.yml">file</a>.</p><p><strong>Key services &amp; why:</strong></p><ul><li><strong>Kafka + Zookeeper</strong> — event backbone(1,2)</li><li><strong>Redis</strong> — fast cache/queues (dedupe, rate limits)</li><li><strong>Neo4j</strong> — knowledge graph</li><li><strong>Elasticsearch + Kibana</strong> — search + quick viz</li><li><strong>api</strong> — FastAPI app</li></ul><p>If you have Docker set up, do the sanity check on the bash terminal:</p><pre># Is docker available?<br>docker --version<br><br>#what is the version of docker-compose? <br>docker compose version || docker-compose version</pre><blockquote>Also, please make sure nothing else is already binding <strong>9093 (host)</strong>, <strong>9092 (internal)</strong>, <strong>2181</strong>, <strong>6379</strong>, <strong>7687</strong>, <strong>7474</strong>, <strong>9200</strong>, <strong>5601</strong>. We need these ports for our services. Sample of docker-compose.yml for zookeeper and kafka are shown as below.</blockquote><pre>services:<br>  zookeeper:<br>    image: confluentinc/cp-zookeeper:7.5.3<br>    environment:<br>      ZOOKEEPER_CLIENT_PORT: 2181<br>      ZOOKEEPER_TICK_TIME: 2000<br>    ports: [&quot;2181:2181&quot;]<br><br>  kafka:<br>    image: confluentinc/cp-kafka:7.5.3<br>    depends_on: [zookeeper]<br>    environment:<br>      KAFKA_BROKER_ID: 1<br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181<br>      # internal + host listeners<br>      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:9093<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9093<br>      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT<br>      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT<br>      # dev-friendly<br>      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1<br>      KAFKA_AUTO_CREATE_TOPICS_ENABLE: &quot;true&quot;<br>      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0<br>    ports: [&quot;9093:9093&quot;]<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;bash&quot;, &quot;-lc&quot;, &quot;kafka-broker-api-versions --bootstrap-server localhost:9092 &gt;/dev/null 2&gt;&amp;1&quot;]<br>      interval: 10s<br>      timeout: 5s<br>      retries: 20</pre><p>Ensure that you copy the code from the GitHub repository. Check whether you have the basic skeleton of folders (apps/ingest, scripts, tests, infra/neo4j)</p><p>Create a .env file. Copy the contents of the file <em>.env.example </em>to <em>.env</em> (Ensure you have Neo4j password added. Also, check whether you need to change any ports. These are our environment settings.</p><pre>cp .env.example .env</pre><p>Ensure that you have a requirements.txt file in your root directory. If not, create one. If you have installed packages manually in your environment, you use the freeze command. Make sure you have feedparser, trafilatura, langdetect, kafka-python-ng[crc32c,lz4,snappy,zstd]==2.2.3 and pydantic entries.</p><pre>pip freeze &gt; requirements.txt</pre><blockquote>pro tip: If you have Kafka installed, uninstall it and replace it with kafka-python. I had to install a fork of kafka-python (kafka-python-ng) to be compatible with other packages. For details, check Kafka documentation.</blockquote><p><strong>The contents of apps/</strong></p><ul><li><strong>apps/ingest/models.py: </strong>Basic structure of the documents received.</li><li><strong>apps/ingest/normalizer.py</strong>: Contains utility functions for text extraction, language detection, etc.</li><li><strong>apps/ingest/producer.py</strong>: To create a Kafka producer instance. A Kafka Producer is a client application that writes (publishes) messages or records to Kafka topics.</li><li><strong>apps/ingest/rss_worker.py: </strong>This<strong> </strong>file is typically a Python script designed to ingest data from one or more RSS feeds and publish it as a stream of events to a Kafka topic (add as many feeds as you want). We used a STUB for checking the initial setup.</li><li><strong>apps/ingest/Dockerfile</strong>: A text file that contains all the commands needed to build a Docker image.</li></ul><p><strong>Topic helper</strong></p><ul><li><strong>scripts/create_topics.sh</strong>: The purpose of this file is to create the topics the pipeline expects with the right names, partitions, and replication. It creates <em>raw-documents</em>, <em>error-documents</em>, and <em>enriched-documents</em>. (There is a version issue with ‘-’ and ‘.’ in the document names. Please follow the requirements of the version. )</li></ul><p><strong>Finally, start services and get ready to ingest</strong></p><ul><li>Step 1:</li></ul><pre># 0) Kafka must be up &amp; healthy<br>docker-compose up -d zookeeper kafka<br><br># 1) Run the script<br>chmod +x scripts/create_topics.sh<br>bash scripts/create_topics.sh</pre><ul><li>Step 2:</li></ul><pre>#!/usr/bin/env bash<br>set -euo pipefail<br><br>BROKER=&quot;kafka:9092&quot;<br><br>echo &quot;Creating Kafka topics (check for dot or hyphen names)&quot;<br>docker-compose exec -T kafka kafka-topics --bootstrap-server &quot;$BROKER&quot; \<br>  --create --if-not-exists --topic raw-documents --partitions 3 --replication-factor 1 || true<br><br>docker-compose exec -T kafka kafka-topics --bootstrap-server &quot;$BROKER&quot; \<br>  --create --if-not-exists --topic enriched-documents --partitions 3 --replication-factor 1 || true<br><br>docker-compose exec -T kafka kafka-topics --bootstrap-server &quot;$BROKER&quot; \<br>  --create --if-not-exists --topic errors-documents --partitions 1 --replication-factor 1 || true<br><br>echo &quot;Topics:&quot;<br>docker-compose exec -T kafka kafka-topics --bootstrap-server &quot;$BROKER&quot; --list</pre><p>Step 3: Verify for the producer</p><pre># Leaders &amp; partitions should show up<br>docker-compose exec -T kafka kafka-topics \<br>  --bootstrap-server kafka:9092 --describe --topic raw-documents<br><br># Produce/consume smoke test<br>echo &#39;{&quot;ping&quot;:&quot;pong&quot;}&#39; | docker-compose exec -T kafka \<br>  bash -lc &#39;kafka-console-producer --broker-list kafka:9092 --topic raw-documents &gt;/dev/null&#39;<br><br>docker-compose exec -T kafka kafka-console-consumer \<br>  --bootstrap-server kafka:9092 --topic raw-documents --from-beginning --max-messages 1</pre><blockquote>Beware of common errors:</blockquote><blockquote>1. LEADER_NOT_AVAILABLE: broker not ready → wait for healthcheck, then re-run. 2. “Nothing printed”: topic exists but empty → produce a test message as above. 3.TTY error when piping: always use -T with docker-compose exec.</blockquote><ul><li>Step 4: Bring up the stack</li></ul><pre># Build images (ingest at minimum)<br>docker-compose build --no-cache ingest<br><br># Start Kafka+ZK and wait for healthy<br>docker-compose up -d zookeeper kafka<br>docker-compose ps   # wait until kafka shows &quot;healthy&quot;<br><br># (optional) start other infra<br>docker-compose up -d redis neo4j elasticsearch kibana<br><br># Create topics (optional if auto-create is ON)<br>bash scripts/create_topics.sh<br><br># Start ingest (use stub first to prove end-to-end)<br>USE_STUB_FEEDS=1 docker-compose up -d --force-recreate ingest<br>docker-compose logs -f --tail=200 ingest</pre><ul><li>You should see output like,</li></ul><p><em>[ingest] rss_worker starting…<br>[ingest] sent &lt;id&gt; -&gt; raw-documents:0@&lt;offset&gt;</em></p><ul><li>Step 5: Simultaneously, you can read the messages in a separate bash terminal in a blocking way</li></ul><pre># read a couple (timeout avoids hanging)<br>docker-compose exec -T kafka kafka-console-consumer \<br>  --bootstrap-server kafka:9092 \<br>  --topic raw-documents \<br>  --from-beginning \<br>  --timeout-ms 5000</pre><ul><li>How to switch to real feeds?</li></ul><pre>USE_STUB_FEEDS=0 docker-compose up -d --force-recreate ingest<br>docker-compose logs -f --tail=200 ingest</pre><p><strong>What does “success” look like?🥳 🎉</strong></p><pre>[ingest] sent &lt;id&gt; -&gt; raw-documents:&lt;partition&gt;@&lt;offset&gt;</pre><p>So, this is all for this post. Let’s see how to enrich the documents in the next post — <strong>Enrichment using NER.</strong></p><p><strong>Additional info: The Event-driven, multi-agent pipeline with a Kafka backbone</strong></p><p>An event-driven, multi-agent pipeline with a Kafka backbone is an architectural pattern for building highly scalable and resilient data processing systems (1). Apache Kafka is a distributed event streaming platform that serves as the central nervous system of the pipeline (2). Instead of tasks being initiated by a central scheduler, the Event-Driven Architecture allows initiation by the arrival of a new event in a Kafka topic (highly reactive to real-time data). The multi-agent pipeline refers to the series of specialized “agents” or microservices that perform specific tasks.</p><p><strong>References (and links):</strong></p><ol><li><a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#:~:text=ZooKeeper%20stores%20its%20application%20configuration,in%20the%20zk%2D0%20Pod">https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#:~:text=ZooKeeper%20stores%20its%20application%20configuration,in%20the%20zk%2D0%20Pod</a>.</li><li><a href="https://medium.com/@turkishtechnology/a-deep-dive-into-distributed-messaging-system-kafka-39304155377c">https://medium.com/@turkishtechnology/a-deep-dive-into-distributed-messaging-system-kafka-39304155377c</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3305e2b40eba" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>