<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gaurav Kumar Vishvakarma on Medium]]></title>
        <description><![CDATA[Stories by Gaurav Kumar Vishvakarma on Medium]]></description>
        <link>https://medium.com/@gauravkrvish?source=rss-2eb74bda54fe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*l12gvehG0aD1sw_JfN19HQ.jpeg</url>
            <title>Stories by Gaurav Kumar Vishvakarma on Medium</title>
            <link>https://medium.com/@gauravkrvish?source=rss-2eb74bda54fe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:22:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gauravkrvish/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build an AI-Powered HR Analytics Assistant Using Python, SQL and GenAI]]></title>
            <link>https://gauravkrvish.medium.com/build-an-ai-powered-hr-analytics-assistant-using-python-sql-and-genai-d8ca3f8eee7e?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8ca3f8eee7e</guid>
            <category><![CDATA[hranalytics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 03:31:01 GMT</pubDate>
            <atom:updated>2026-11-10T03:31:01.835Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uzVi-od54ul_-TlXpMrs_w.png" /></figure><p>Most HR teams have the same Monday problem. Someone senior asks a simple question, &quot;what’s our attrition right now?&quot; or &quot;why is hiring in engineering so slow?&quot;, and the answer takes two days. Not because the question is hard, but because the data sits in three systems, the definitions live in someone’s head, and the person who builds the report is already busy with five other reports.</p><p>Dashboards help, but only for the questions someone thought of in advance. The follow-up question is where it breaks down. So I built a small assistant that lets an HR manager type a question in plain English and get back a number, a chart and the exact SQL behind it, without ever touching the raw employee table.</p><p>This post walks through the whole thing: the KPIs, the database design, the SQL, the GenAI layer, the Streamlit app and, most importantly, the privacy guardrails. All the data is synthetic (no real employees), generated with Python. The stack is <strong>PostgreSQL, Python, an LLM API and Streamlit</strong>, and the whole project is about 400 lines of code.</p><h3>1. What is HR analytics, really?</h3><p>HR analytics is using workforce data to answer business questions about people: who is leaving and why, how long it takes to fill a role, where candidates drop out, whether the organisation is growing or shrinking. It’s not about having an HR dashboard. It’s about being able to answer the next question quickly and with numbers everyone trusts.</p><p>The trust part matters more than people expect. If finance calculates attrition one way and HR another, the meeting turns into an argument about the number instead of a decision about the problem.</p><h3>2. Why HR reporting is still painful</h3><p>The challenges are boringly consistent across companies:</p><ul><li><strong>Data is scattered.</strong> HRMS for employees, an ATS for recruitment, spreadsheets for everything in between.</li><li><strong>Definitions drift.</strong> Is attrition based on starting headcount or average headcount? Do internal transfers count as exits? Ask three people and you get three answers.</li><li><strong>Everything is sensitive.</strong> Salaries, performance ratings, exit reasons and personal details mean you can’t just give everyone database access.</li><li><strong>The report queue never ends.</strong> Every ad-hoc question becomes a ticket to the one analyst who knows the data.</li></ul><p>An AI assistant can help with the last one. But it only works if you fix the first three first. That’s really what this project is about.</p><h3>3. How the assistant works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*juq2MaRHjNS0trKFC4logw.png" /></figure><p>The flow is short:</p><ol><li>The HR manager asks a question in a Streamlit chat box.</li><li>If it matches a question we’ve already approved, the assistant uses vetted SQL. Otherwise the LLM writes a query, using only a list of approved views and written KPI rules.</li><li>A validator checks the SQL before it runs: one SELECT statement, approved views only, a row limit.</li><li>The query runs on PostgreSQL under a read-only role that can’t even see the raw tables.</li><li>The app shows a short answer, a chart, the data and the SQL used.</li></ol><p>The LLM never sees employee rows. It sees a schema description and, at the end, a small aggregated result to summarise.</p><h3>4. Designing the employee dataset</h3><p>I generated a synthetic company with Python: seven departments (Sales, Customer Support, Engineering, Operations, Finance, HR and Marketing), about 1,000 employee records from 2017 to October 2026, 409 job requisitions and 13,850 candidate applications. Each department has its own attrition tendency, and junior levels leave more often, so the patterns look like a real company instead of random noise.</p><p>Three tables are enough to start:</p><pre>CREATE TABLE employees (<br>    employee_id        INT PRIMARY KEY,<br>    department         TEXT NOT NULL,<br>    job_level          TEXT NOT NULL,<br>    location           TEXT,<br>    hire_date          DATE NOT NULL,<br>    exit_date          DATE,<br>    exit_reason        TEXT,<br>    performance_rating INT<br>);<br><br>CREATE TABLE requisitions (<br>    requisition_id INT PRIMARY KEY,<br>    department     TEXT NOT NULL,<br>    opened_date    DATE NOT NULL,<br>    filled_date    DATE,<br>    status         TEXT NOT NULL      -- Open / Filled<br>);<br><br>CREATE TABLE candidates (<br>    candidate_id   INT PRIMARY KEY,<br>    requisition_id INT REFERENCES requisitions(requisition_id),<br>    department     TEXT,<br>    source         TEXT,<br>    applied_date   DATE,<br>    stage_reached  TEXT,              -- Applied, Screened, Interviewed, Offered, Hired<br>    offer_made     BOOLEAN,<br>    offer_accepted BOOLEAN<br>);</pre><p>Notice what’s missing: names, emails, phone numbers, salaries. An analytics assistant doesn’t need them to answer workforce questions, so they don’t belong in this database at all. That one decision removes most of the privacy risk before we write any AI code.</p><h3>5. Defining the HR KPIs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iM4TI_L7rbBHIrpEdmiBeg.png" /></figure><p>Before any AI, write the definitions down. These are the five I used, with the numbers from the synthetic data as of 31 October 2026:</p><ul><li><strong>Headcount:</strong> active employees at month-end. Hired on or before that date, and not exited. Current value: <strong>598</strong>.</li><li><strong>Attrition rate:</strong> exits in the last 12 months divided by average monthly headcount over the same 12 months. Current value: <strong>25.4%</strong> (155 exits, average headcount 611).</li><li><strong>Time to hire:</strong> days from requisition opened to requisition filled. Report the median alongside the average, because a few hard-to-fill roles drag the average up. Current value: <strong>46.3 days</strong> on average.</li><li><strong>Offer acceptance rate:</strong> offers accepted divided by offers made. Current value: <strong>73.2%</strong>.</li><li><strong>Recruitment conversion:</strong> share of candidates who move from one stage to the next.</li></ul><p>The attrition definition is the one people argue about most. Dividing by average headcount instead of starting headcount gives a fairer number when the company is growing or shrinking. Whatever you choose, the point is that the assistant uses the same definition every time, and so does the dashboard.</p><h3>6. SQL views for workforce metrics</h3><p>Here’s the design decision that makes the whole thing safe: <strong>the assistant never queries the raw tables.</strong> It queries a small set of views, each one implementing a KPI definition exactly once.</p><p>Month-end headcount, hires and exits per department:</p><pre>CREATE VIEW v_monthly_headcount AS<br>WITH months AS (<br>    SELECT (date_trunc(&#39;month&#39;, d) + interval &#39;1 month - 1 day&#39;)::date AS month_end<br>    FROM generate_series(date &#39;2024-01-01&#39;, date &#39;2026-10-01&#39;, interval &#39;1 month&#39;) d<br>)<br>SELECT m.month_end, e.department,<br>       COUNT(*) FILTER (WHERE e.hire_date &lt;= m.month_end<br>                          AND (e.exit_date IS NULL OR e.exit_date &gt; m.month_end)) AS headcount,<br>       COUNT(*) FILTER (WHERE e.exit_date BETWEEN date_trunc(&#39;month&#39;, m.month_end)::date<br>                                             AND m.month_end)                    AS exits,<br>       COUNT(*) FILTER (WHERE e.hire_date BETWEEN date_trunc(&#39;month&#39;, m.month_end)::date<br>                                             AND m.month_end)                    AS hires<br>FROM months m CROSS JOIN employees e<br>GROUP BY m.month_end, e.department;</pre><p>Rolling 12-month attrition, built on top of it:</p><pre>CREATE VIEW v_attrition_12m AS<br>SELECT department,<br>       SUM(exits)                                    AS exits_12m,<br>       ROUND(AVG(headcount), 1)                      AS avg_headcount_12m,<br>       ROUND(100.0 * SUM(exits) / AVG(headcount), 1) AS attrition_rate_pct<br>FROM v_monthly_headcount<br>WHERE month_end &gt; (SELECT MAX(month_end) FROM v_monthly_headcount) - interval &#39;12 months&#39;<br>GROUP BY department;</pre><p>I added three more views the same way: v_time_to_hire, v_recruitment_funnel and v_offer_acceptance. Then a dedicated database role that can read the views and nothing else:</p><pre>CREATE ROLE hr_assistant LOGIN PASSWORD &#39;change-me&#39;;<br>GRANT USAGE ON SCHEMA public TO hr_assistant;<br>GRANT SELECT ON v_monthly_headcount, v_attrition_12m, v_time_to_hire,<br>               v_recruitment_funnel, v_offer_acceptance TO hr_assistant;<br>ALTER ROLE hr_assistant SET default_transaction_read_only = on;<br>ALTER ROLE hr_assistant SET statement_timeout = &#39;5s&#39;;</pre><p>If the assistant tries SELECT * FROM employees, PostgreSQL itself says <em>permission denied</em>. That’s a guardrail that doesn’t depend on the AI behaving well.</p><h3>7. Connecting the data to Python</h3><p>The Python side uses SQLAlchemy and pandas, connecting as the read-only role:</p><pre>from sqlalchemy import create_engine, text<br>import pandas as pd<br><br>DB_URL = &quot;postgresql://hr_assistant:change-me@localhost:5432/hr&quot;<br><br>def run(sql: str) -&gt; pd.DataFrame:<br>    engine = create_engine(DB_URL)      # read-only role, 5-second timeout<br>    with engine.connect() as con:<br>        return pd.read_sql(text(sql), con)</pre><p>In a real company, the connection string comes from a secrets manager, not the code, and the views are refreshed from the HRMS and ATS on a schedule.</p><h3>8. The LLM question-answering layer</h3><p>This is where most tutorials start, and where most of them go wrong. Three things make it reliable.</p><p><strong>First, an approved-question library.</strong> The questions HR asks every week (attrition, headcount, time to hire) get SQL that a human has reviewed. The LLM only writes new SQL for questions that aren’t in the library. Same question, same answer, every time.</p><p><strong>Second, a prompt built from the views and the KPI rules,</strong> not from the raw schema:</p><pre>KPI_RULES = &quot;&quot;&quot;<br>- Attrition rate (company-wide) = SUM(exits_12m) / SUM(avg_headcount_12m) * 100.<br>  Never average department percentages.<br>- Headcount is a month-end snapshot. Company headcount = SUM(headcount) per month_end.<br>- Time to hire = days from requisition opened to filled. Report average AND median.<br>- Offer acceptance rate = accepted / offers * 100.<br>&quot;&quot;&quot;</pre><p>The line about not averaging department percentages is there because it’s the most common mistake, made by humans and models alike. Averaging seven department rates gives a different number from the real company-wide rate.</p><p><strong>Third, a validator that runs before anything touches the database.</strong> I use sqlglot to parse the query and check it:</p><pre>import sqlglot<br>from sqlglot import exp<br><br>def validate_sql(sql: str) -&gt; str:<br>    parsed = [s for s in sqlglot.parse(sql, read=&quot;postgres&quot;) if s is not None]<br>    if len(parsed) != 1 or not isinstance(parsed[0], exp.Select):<br>        raise UnsafeQuery(&quot;Only a single SELECT statement is allowed.&quot;)<br>    tree = parsed[0]<br>    ctes = {c.alias_or_name.lower() for c in tree.find_all(exp.CTE)}<br>    for table in tree.find_all(exp.Table):<br>        if table.name.lower() not in ALLOWED_VIEWS and table.name.lower() not in ctes:<br>            raise UnsafeQuery(f&quot;Table not allowed: {table.name}&quot;)<br>    if not tree.args.get(&quot;limit&quot;):<br>        tree = tree.limit(500)<br>    return tree.sql(dialect=&quot;postgres&quot;)</pre><p>I tested it with the obvious bad inputs: a query against the raw employees table, a delete on one of the views, and a second statement sneaked in after a semicolon. All three are rejected before they reach PostgreSQL. If a generated query fails validation or errors in the database, the error goes back to the model with the original question, and it gets two more attempts. After that, the assistant says it can’t answer instead of guessing.</p><p>The final step is the summary. The model gets the question and the result rows (aggregates only) and is told to use only numbers from the result and not to speculate about causes. For the approved questions I went one step further and used fixed sentence templates, so the wording is as consistent as the numbers.</p><h3>9. Visualisation in Streamlit</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4_cq2BLJjCS6u3wr2TlvJQ.png" /></figure><p>The app is about 50 lines of Streamlit. A KPI strip at the top reads straight from the views. Below it there’s a chat input. For each answer the app picks a chart from the shape of the result: a line for anything by month, a funnel for recruitment stages, a horizontal bar for comparisons by department. Then the table, then a collapsible “SQL used” box that shows whether the query was approved or generated.</p><pre>question = st.chat_input(&quot;Ask a question about your workforce&quot;)<br>if question:<br>    r = answer(question)<br>    st.write(r[&quot;summary&quot;])<br>    df = r[&quot;data&quot;]<br>    if &quot;month&quot; in df.columns[0]:<br>        fig = px.line(df, x=df.columns[0], y=df.columns[1], markers=True)<br>    elif df.columns[0] == &quot;stage&quot;:<br>        fig = px.funnel(df, x=&quot;candidates&quot;, y=&quot;stage&quot;)<br>    else:<br>        fig = px.bar(df.sort_values(df.columns[-1]), x=df.columns[-1],<br>                     y=df.columns[0], orientation=&quot;h&quot;)<br>    st.plotly_chart(fig, use_container_width=True)<br>    st.dataframe(df, hide_index=True)<br>    with st.expander(f&quot;SQL used ({r[&#39;source&#39;]})&quot;):<br>        st.code(r[&quot;sql&quot;], language=&quot;sql&quot;)</pre><p>Showing the SQL is not decoration. It’s how an analyst checks the assistant’s work, and it’s how HR learns to trust it.</p><h3>10. What the assistant found</h3><p>Here are the five example questions and what the assistant returned from the synthetic data:</p><p><strong>“What is the current employee attrition rate?”</strong> 25.4% over the last 12 months: 155 exits against an average headcount of 611.</p><p><strong>“Which department has the highest attrition?”</strong> Customer Support at 42.7% (39 exits), followed by Marketing at 35.4%. Finance is lowest at 10.3%. Customer Support alone accounts for a quarter of all exits, which is where an HR business partner would start.</p><p><strong>“What is the average time to hire?”</strong> 46.3 days across 372 filled roles. Engineering is the outlier at 62.6 days on average (median 58.5), while Operations fills roles in 37.3 days.</p><p><strong>“How has headcount changed over six months?”</strong> From 612 in May 2026 to 598 in October 2026, down 14. There were 76 exits and only 62 hires, and September was the worst month: 18 exits, 6 hires.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNw5jDH9jFjoT_OYlCc7Lg.png" /></figure><p><strong>“Which recruitment stage has the highest drop-off?”</strong> This one is more interesting than it looks. The biggest <em>count</em> drop is at screening: 13,850 applicants down to 6,548. But the biggest <em>rate</em> drop is from interview to offer, where only 17.9% move on (2,832 to 508). That points to a different problem: either interviewers are rejecting most candidates, or the wrong people are reaching interviews. The assistant gives you the number. Deciding which of those it is still needs a person.</p><h3>11. Privacy and responsible AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y139Tmt0JmWy0zq4jBhzLw.png" /></figure><p>HR data is the most sensitive data most companies hold, so this section isn’t optional. The guardrails in this build:</p><ul><li><strong>No personal identifiers in the analytics database.</strong> No names, contact details or salaries. If you don’t store it, it can’t leak.</li><li><strong>Views, not tables.</strong> The assistant’s database role can read five aggregate views and nothing else, enforced by PostgreSQL.</li><li><strong>The LLM never sees individual rows.</strong> It gets the schema description and aggregated results only.</li><li><strong>Small-group suppression.</strong> In production, hide any result where a group has fewer than five people. “Attrition in the two-person legal team” identifies individuals.</li><li><strong>Everything is logged.</strong> Question, SQL, approved or generated, who asked and when.</li><li><strong>The assistant reports, it doesn’t judge.</strong> It doesn’t predict who will resign or rate individual employees. Decisions about people stay with people.</li></ul><p>If you’re in India, the DPDP Act and Rules apply to employee data too, including purpose limitation and keeping processing logs. If you have European employees, GDPR applies. Using an LLM provider means checking where data is processed and that it isn’t used for training. I wrote more about this in <a href="https://medium.com/p/661e8fcfdc86">AI Data Governance: Why Most AI Problems Are Really Data Problems</a>.</p><h3>What I’d add next</h3><ul><li>Real connectors to the HRMS and ATS instead of synthetic data, refreshed nightly.</li><li>Row-level security, so a department head only sees their own department.</li><li>A feedback button on each answer, so wrong answers get reviewed and good generated queries can be promoted into the approved library.</li><li>A Power BI dashboard on the same views, so the dashboard and the assistant always agree.</li></ul><h3>The takeaway</h3><p>The AI part of this project is maybe 20% of the code. The rest is the unglamorous work: agreeing KPI definitions, building clean views, locking down access and deciding what the assistant is <em>not</em> allowed to do. That’s also why it works. An HR manager can ask a question on Monday morning and get a trustworthy answer in seconds, with the SQL right there if anyone wants to check.</p><p>If you’re building something similar, these go deeper on the pieces:</p><ul><li><a href="https://gauravkrvish.medium.com/text-to-sql-explained-with-practical-examples-and-where-it-quietly-goes-wrong-96599835f40a">Text-to-SQL Explained with Practical Examples (and Where It Quietly Goes Wrong)</a></li><li><a href="https://gauravkrvish.medium.com/i-built-an-ai-data-analyst-from-natural-language-to-sql-to-business-insights-831539717fa3">I Built an AI Data Analyst: From Natural Language to SQL to Business Insights</a></li><li><a href="https://gauravkrvish.medium.com/hr-analytics-isnt-about-dashboards-it-s-about-the-questions-nobody-could-answer-553391dc6a7a">HR Analytics Isn’t About Dashboards. It’s About the Questions Nobody Could Answer.</a></li><li><a href="https://gauravkrvish.medium.com/how-i-built-an-hr-analytics-dashboard-in-power-bi-a-step-by-step-guide-69dafef0a7c7">How I Built an HR Analytics Dashboard in Power BI</a></li></ul><p>I build HR and business dashboards, data pipelines and practical AI tools for teams. You can see my work at <a href="https://iamgaurav.netlify.app/">iamgaurav.netlify.app</a>.</p><p><em>All data in this article is synthetic and generated with Python. No real employee or candidate data was used.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8ca3f8eee7e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Agent Analytics: When Several AI Agents Beat One, and When They Just Cost More]]></title>
            <link>https://gauravkrvish.medium.com/multi-agent-analytics-when-several-ai-agents-beat-one-and-when-they-just-cost-more-fc0166d15c4c?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc0166d15c4c</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 03:31:01 GMT</pubDate>
            <atom:updated>2026-11-10T03:31:01.669Z</atom:updated>
            <content:encoded><![CDATA[<p>The most dangerous thing a data assistant can do is hand you a wrong number with complete confidence. A chart renders, the figure looks plausible, and nobody questions it until someone in finance asks why revenue is two and a half times last quarter&#39;s.</p><p>It is also a topic with a lot of hype around it, so this article tries to stay practical. I look at four public data points, two in favour and two urging caution, build a small example where one agent catches another one&#39;s mistake, and finish with a short test for deciding whether you need several agents at all.</p><p>A note on the example: it runs on a synthetic orders table I generated, and the agents in it are plain Python functions standing in for model calls. The point is the structure, not the model. Every number in that section comes from that fake data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LbsusJfCbfULjnxjKPdVKQ.png" /></figure><h3>What Multi-Agent Analytics Actually Means</h3><p>Strip away the vocabulary and a multi-agent analytics system is a workflow where more than one AI model, each with its own instructions and its own tools, works on a single question. Each agent has a narrow job. They pass work to each other, and usually one of them, often called the lead or the orchestrator, decides who does what.</p><p>That is different from a single agent, which runs one loop: it plans, queries, looks at the result and decides what to do next, all inside one shared conversation. A single agent is simpler and cheaper. Its weakness is that it also marks its own homework. If it misreads a table in step two, it tends to carry that misreading all the way to the answer.</p><p>The case for several agents rests on two things. One is parallelism: independent sub-questions can be worked on at the same time. The other, which matters more in analytics, is independence: a second agent that approaches the problem from a different angle can catch what the first one cannot see.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ubimPdDBhoBecXXhaUV5Bw.png" /></figure><h3>The Roles I Keep Seeing</h3><p>Whatever the product, the cast is surprisingly consistent. A router decides what kind of request has arrived. A schema agent finds the right tables and fields in a warehouse that may hold thousands of them. A writer drafts and revises the SQL. A validator checks the query and its output. An analyst turns the rows into a plain-language answer, and sometimes a critic challenges that answer before it goes out.</p><p>You do not need all six. If I were starting a project tomorrow, I would build a writer and a validator first, because that pair addresses the failure that hurts most: a confident wrong number. Everything else can be added when a real problem asks for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gjGJ6fHCUSbiKA7Y7MGCvA.png" /></figure><h3>Four Public Data Points</h3><p>I wanted evidence that did not come from a vendor pitch, so I collected four public sources: two that describe multi-agent systems working, and two that argue for restraint. I have kept to what each one actually says.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c1g8wWIjYGxck6K1Xd54Rg.png" /></figure><h3>Case 1: Anthropic&#39;s Multi-Agent Research System</h3><p>Anthropic described how it built the multi-agent research feature in Claude. A lead agent plans the work and starts several subagents that search in parallel, then it pulls their findings together. A separate agent adds citations at the end. On an internal research evaluation, the version with Claude Opus 4 as lead and Claude Sonnet 4 subagents scored 90.2 percent better than a single Claude Opus 4.</p><p>The more useful part is what came with that number. Anthropic reports that multi-agent systems used about 15 times more tokens than a normal chat, and that token usage alone explained around 80 percent of the performance differences on one benchmark. In plain terms, part of the improvement was simply that more work was being done. The early versions also went wrong in familiar ways: starting 50 subagents for a simple question, searching endlessly for sources that did not exist, and agents duplicating each other&#39;s work when instructions were vague.</p><p>That last point stuck with me. The fix was mostly better delegation: each subagent needed a clear objective, an output format and boundaries. Anyone who has handed a vague task to a new colleague will recognise it.</p><h3>Case 2: LinkedIn&#39;s SQL Bot</h3><p>The reported results are modest and believable. About 95 percent of surveyed users rated the accuracy of the queries as passing or better, and roughly 40 percent rated it very good or excellent. The team also noted that building it into the tools people already used raised adoption by five to ten times compared with a standalone chatbot. And they observed that validators help most when they check something the query writer does not have access to, which is exactly the idea behind the example later in this article.</p><h3>Case 3: The Argument Against, From Cognition</h3><p>Cognition, the company behind the Devin coding agent, published an essay titled Don&#39;t Build Multi-Agents. Its argument is that parallel agents cannot see each other&#39;s decisions, so they make conflicting assumptions. Its example is a task split between two subagents, where one built a background in one visual style and the other built a character in a different one, and a final agent had to merge two things that did not fit.</p><p>The essay recommends a single linear agent that carries the full context, with a separate model to compress long histories when needed. I read this as a warning rather than a rejection. Work with many dependencies between steps, which describes a lot of analytics, is where splitting agents costs the most. Anthropic&#39;s own write-up says something similar: multi-agent systems suit tasks that can be parallelised and fit poorly where all the agents need the same shared context.</p><h3>Case 4: Gartner&#39;s Forecast</h3><p>In June 2025 Gartner predicted that over 40 percent of agentic AI projects will be canceled by the end of 2027, citing rising costs, unclear business value and inadequate risk controls. It also estimated that only about 130 of the thousands of vendors selling agentic AI are genuinely delivering it, with many relabelling chatbots and automation tools as agents.</p><p>That is a forecast, not a measurement, so I hold it loosely. But the three reasons it gives are the same ones I would worry about in an analytics team: the bill, the value and the controls.</p><h3>What the Four Cases Agree On</h3><p>Read together, they point the same way. Multi-agent designs can improve results, but the improvement is paid for in tokens and in complexity. The wins reported by Anthropic and LinkedIn both include an independent step that checks the work: a citation agent in one case, validators in the other. The cautions from Cognition and Gartner are about shared context, cost and unclear value.</p><p>None of them says never use several agents. All of them say to be sure what the extra agents are for. Which brings me to the part I could test myself.</p><h3>Build It: A Checker That Catches a Bad Join</h3><p>Here is the example. I made a small SQLite database with 2,000 fake orders and an order_items table holding one to four line items per order. The question is simple: revenue by region for Q3 2026.</p><p>Three agents are involved. The writer stands in for a model that drafts SQL. The checker is deliberately independent: it never sees the writer&#39;s query, it just runs its own plain total for the same period. The orchestrator compares the two and, if they disagree, sends the question back to the writer with feedback. In a real system the writer would be a model call. Here it is a function, so you can run the whole thing and see the logic.</p><p>The writer&#39;s first draft does what plenty of people do on a bad day. It joins orders to order_items to be thorough, then sums the order amount. This is the query:</p><pre>SELECT o.region, ROUND(SUM(o.amount), 2) AS revenue<br>FROM orders o JOIN order_items i ON i.order_id = o.order_id<br>WHERE o.order_date BETWEEN &#39;2026-07-01&#39; AND &#39;2026-09-30&#39;<br>GROUP BY o.region</pre><p>The checker&#39;s own query is just the sum of amount from orders for the same dates, with no join. The orchestration around them is a short loop:</p><pre>def validate(rows):<br>    got = round(sum(r[1] for r in rows), 2)<br>    want = control_total()   # separate, simple query<br>    return abs(got - want) &lt; 0.01, got, want<br><br>sql = writer(question)<br>for attempt in (1, 2):<br>    rows = db.execute(sql).fetchall()<br>    ok, got, want = validate(rows)<br>    if ok:<br>        break<br>    sql = writer(question, feedback=&quot;total does not reconcile&quot;)</pre><p>On the first attempt the writer&#39;s total came back as 1,069,375 dollars. The checker&#39;s control total was 424,156. The ratio was 2.52, which is close to the average number of line items per order, and that points straight at the cause: the join repeated each order once per item, so every amount was summed several times. The orchestrator rejected the result and sent it back with feedback. The second draft, which dropped the unnecessary join, matched the control total to the cent.</p><p>Notice what made this work. The checker did not review the writer&#39;s SQL, which would have meant spotting a subtle flaw by reading. It compared two independent routes to the same number. When they disagree, you do not need to know why to know the answer is not ready to be shown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfaaEM4auDmQubdxYbqccQ.png" /></figure><h3>Where the Checker Earns Its Keep</h3><p>This pattern is cheap and worth borrowing even if you never build anything bigger. A total reconciliation catches fan-out joins, duplicated rows and wrong date filters. A row-count check catches a join that quietly dropped data. A comparison with last month&#39;s published figure catches a definition that has drifted. None of these needs a clever model. They need something that was not involved in writing the query.</p><p>It has limits. A checker only catches errors it can measure. If the writer and the checker share the same wrong definition of revenue, they will agree with each other and both be wrong. Independent does not mean correct.</p><h3>When Not to Use Several Agents</h3><p>Most analytics questions do not need a team. Ask for last month&#39;s sales by product and a single agent with a good schema description will do it faster and for a fraction of the cost. Multiple agents make sense when the question breaks into independent parts, when a wrong answer is expensive, or when an independent check has a good chance of catching real errors.</p><p>Cost is the other half. Anthropic&#39;s figure of about 15 times the tokens of a chat means every extra agent should have to justify itself. Before building anything, I would run through the four questions below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZlCodT551ECNLkcz5gsj2g.png" /></figure><h3>Guardrails I Would Put In First</h3><p>Analytics agents write queries against real data, so the guardrails matter more than the architecture. Give the SQL agent a read-only database role so it cannot change anything. Cap row counts and query time. Log every agent&#39;s input and output, so that when a number is wrong you can see which step went wrong. Keep a human in the loop for anything that leaves the building, such as a figure going into a board pack.</p><p>Evaluation deserves the same attention. LinkedIn&#39;s team keeps a benchmark of real questions with accepted answers and reviews it regularly. Without something like that, you cannot tell whether a change to a prompt made the system better or only different.</p><h3>Where to Start</h3><p>If multi-agent analytics interests you, I would start smaller than the diagrams suggest. Build one agent that writes SQL against a well-described schema. Add an independent check that reconciles totals. Log everything for a few weeks and read what goes wrong. Add a second or third agent only when you can name the specific failure it will fix.</p><p>The best version of this I can imagine is not a swarm. It is a quiet system that is right more often than a tired analyst on a Friday afternoon, and that says so when it is not sure.</p><p>If you are working on something similar and want a second opinion, I write about Power BI, SQL and practical analytics, and you can find more of my work at <a href="https://iamgaurav.netlify.app/">iamgaurav.netlify.app</a>.</p><h3>Sources</h3><p>The example and its numbers come from a synthetic dataset I generated; everything else below is public.</p><ul><li>Anthropic Engineering, How we built our multi-agent research system: https://www.anthropic.com/engineering/multi-agent-research-system</li><li>ZenML LLMOps database, LinkedIn&#39;s multi-agent text-to-SQL assistant: https://www.zenml.io/llmops-database/building-a-production-text-to-sql-assistant-with-multi-agent-architecture</li><li>Cognition, Don&#39;t Build Multi-Agents: https://cognition.com/blog/dont-build-multi-agents</li><li>Gartner press release, over 40% of agentic AI projects will be canceled by end of 2027: https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027</li><li>SD Times, Gartner: more than 40% of agentic AI projects will be canceled: https://sdtimes.com/agentic-ai/gartner-more-than-40-of-agentic-ai-projects-will-be-canceled-in-next-few-years/</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc0166d15c4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI-Powered HR Analytics: Six Real Case Studies and a Model You Can Actually Build]]></title>
            <link>https://gauravkrvish.medium.com/ai-powered-hr-analytics-six-real-case-studies-and-a-model-you-can-actually-build-b0be195638ec?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0be195638ec</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[hranalytics]]></category>
            <category><![CDATA[power-bi]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 05:28:31 GMT</pubDate>
            <atom:updated>2026-11-09T05:28:31.421Z</atom:updated>
            <content:encoded><![CDATA[<p>&quot;Can AI tell us who is about to quit?&quot; It is the question I hear most often from HR teams, usually in the first ten minutes of a call. The honest answer is: partly, and what matters far more than the model is what you do with the answer.</p><p>HR is one of the most promising places to apply analytics and one of the riskiest. The data describes people, the decisions affect livelihoods, and several well-known AI projects in this space have failed in public. Hiring tools were scrapped, vendors dropped features after complaints, and regulators and courts have started to pay attention.</p><p>This article is the long version of how I think about it. It covers six real, publicly documented case studies, the architecture I would use, a working attrition model with honest numbers, and the fairness checks that most projects skip. Every case study is linked at the end so you can read the sources yourself.</p><p>One note on the build section: it uses a synthetic dataset I generated myself. Real employee data has no business in a blog post, so every number there comes from fake people, and I label it that way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WiWhzHtjUlQXUGb98-76aQ.png" /></figure><h3>What HR Analytics Actually Is</h3><p>HR analytics means using workforce data to answer people questions with evidence instead of instinct. It helps to think of it in four levels, because the level you are on decides which tools make sense.</p><p>The first is descriptive: headcount, attrition rate, time to hire, absence. It answers what happened, and it is dashboard territory. The second is diagnostic: why did it happen? Attrition by manager, by tenure band, by pay position. The third is predictive: which teams or roles are likely to lose people next? The fourth is prescriptive: given all that, what should we actually change?</p><p>In my experience, most HR teams are stuck between the first and second level, and AI is not the reason. Their data sits in an HR system, an applicant tracking system, a payroll file and two or three survey tools that do not share a common employee ID. Joining those cleanly is the dull part nobody puts in a demo, and it is worth more than any model built on top of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5mWyNwcXydBlbRRUmK2Rtw.png" /></figure><p>Where AI Genuinely Helps in HR</p><p>Before the case studies, it is worth being specific about where AI earns its place. The pattern I trust is simple: use AI where it saves people from reading and sorting, and keep humans where a decision affects someone&#39;s job, pay or career.</p><p>The uses I see working are team-level attrition risk, so HR knows where to look first; headcount and hiring forecasts; analysis of thousands of free-text survey comments, where a language model can group themes in minutes instead of weeks; answering routine policy questions through an HR assistant; and matching internal employees to open roles based on skills.</p><p>The uses that keep ending up in the news are the other kind: automatically scoring candidates, ranking employees, or flagging individuals as flight risks without anyone checking. The six cases below mostly come from that second group, and they are instructive precisely because the people behind them were not careless. Most were well funded and well intentioned.</p><h3>Six Real Case Studies</h3><p>Everything in this section is public record. Where a figure comes from a company&#39;s own statements, I say so, because a claim is not the same as a finding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0AYilAs0Y3I9tbTcODICxg.png" /></figure><p>Case 1: Google and the Question Nobody Expected</p><p>Google&#39;s People Analytics team set out to test an assumption many engineers held: that managers do not matter much. They studied what separated the highest-performing teams from the rest and published the findings as a list of eight habits of effective managers. Being a good coach and not micromanaging came near the top. So did showing interest in the team&#39;s success and wellbeing, communicating clearly, and helping people with their careers.</p><p>The result was not a prediction. It was a set of behaviours a manager could change, and a clear finding that manager quality was one of the biggest factors the company could control. That is the case I point to when someone assumes HR analytics has to mean machine learning. The most valuable output was an answer to a good question, delivered in a form people could act on.</p><h3>Case 2: Amazon&#39;s Resume Screener</h3><p>Amazon&#39;s machine learning team in Edinburgh began building a tool in 2014 that scored job applicants from one to five stars, much like a product review. It was trained on resumes submitted to the company over ten years. Most of those resumes came from men, which reflected who worked in tech, and the model learned the pattern.</p><p>By 2015 the team had realised the system was penalising resumes that contained the word &quot;women&#39;s&quot;, as in a women&#39;s chess club captain, and downgrading graduates of two all-women&#39;s colleges. Amazon edited the system to be neutral to those specific terms, but, as Reuters reported, that was no guarantee the model would not find other ways to sort candidates that were just as discriminatory. The team was disbanded around the start of 2017. Recruiters had looked at its recommendations, but according to Reuters&#39; sources they never relied on the rankings alone.</p><p>The lesson is not that AI is biased by nature. It is that a model trained on historical decisions learns whatever those decisions rewarded, including things you would never put in a policy. You cannot fix that by deleting one keyword.</p><h3>Case 3: IBM and the 95 Percent Claim</h3><p>In 2019 IBM&#39;s chief executive Ginni Rometty said the company&#39;s predictive attrition program, built with Watson, could identify employees likely to leave with an accuracy in the 95 percent range, and that it had saved IBM nearly 300 million dollars in retention costs. IBM holds a patent on the approach, and the system reportedly suggests actions managers can take to re-engage people at risk. She did not say which data points it used or how the figures were calculated.</p><p>I include it because it shows how to read a headline accuracy number. If about 15 percent of employees leave in a year, a model that predicts that nobody will leave is 85 percent accurate and completely useless. The questions worth asking of any vendor are accuracy on what, measured when, against what baseline, and how many of the people flagged actually left. None of that was public here. It may be a very good system. We simply cannot tell from the claim.</p><h3>Case 4: HireVue and the Face Problem</h3><p>HireVue&#39;s video interview software once analysed candidates&#39; facial expressions as one input to its assessments. In 2019 the Electronic Privacy Information Center filed a complaint with the US Federal Trade Commission about the practice, and by early 2021 the company had dropped it. HireVue&#39;s chief data scientist said the visual signals contributed only about 0.25 percent to a model&#39;s predictive power.</p><p>The company also commissioned an outside algorithmic audit, which mostly gave good marks but recommended looking at how the system handled different accents, and at very short answers, which minority candidates gave more often and the system found hard to score. Two takeaways: a feature that adds almost nothing but creates real risk should be cut, and an independent audit finds things the builders miss.</p><h3>Case 5: New York City Local Law 144</h3><p>Not every case study is a failure story. New York City&#39;s Local Law 144 requires employers that use automated tools for hiring or promotion decisions to commission an independent bias audit every year, publish a summary of the results, and notify candidates that such a tool is being used. Penalties can reach 1,500 dollars per violation per day.</p><p>A December 2025 audit by the New York State Comptroller found that enforcement had been weak. In test calls to the city&#39;s 311 line about these tools, 75 percent were misrouted and never reached the enforcing department. The department&#39;s own review of 32 published bias audits found one problem, while the Comptroller&#39;s staff found at least 17 potential issues in the same set. That is a story about immature enforcement, not about the law being harmless. The direction of travel is clear, and a requirement that is lightly enforced today will not stay that way.</p><h3>Case 6: Mobley v. Workday</h3><p>Mobley v. Workday is a federal case in California in which the plaintiff alleges that Workday&#39;s AI-based applicant recommendation system had a discriminatory impact on applicants based on race, age and disability. Notably, he does not claim Workday was his employer. He argues that Workday can be liable as an agent of the employers who use its software.</p><p>The court let the case proceed in 2024, and in May 2025 it conditionally certified the age-discrimination claim as a collective action, finding that the plaintiff had pointed to a single policy: the use of Workday&#39;s recommendation system to score, sort, rank or screen applicants. Workday has stated that 1.1 billion applications were rejected using its tools during the relevant period, which is why the potential group is so large. These are allegations, the case has not been decided, and I am not a lawyer. But the practical message for any HR team is already clear: buying a vendor&#39;s tool does not outsource your responsibility for what it does.</p><h3>What the Six Cases Have in Common</h3><p>Reading them together, a few patterns repeat. The successful project started from a question and ended with something people could act on. The failures started from a technology and went looking for a use. The claims that could not be checked were the ones with no published method. And in every case where people were being scored, someone eventually asked who was checking the system for fairness, and the honest answer was often nobody.</p><p>None of this means HR should avoid analytics. It means the useful parts, understanding why people leave, where workload is unsustainable and which managers need support, are mostly the unglamorous ones. Which brings me to what I would actually build.</p><h3>The Stack Behind a Responsible HR Project</h3><p>The architecture is almost boring, and that is the point. Data flows from the HR system, applicant tracking and survey tools into a warehouse, where it is cleaned and joined into one model. Employee identifiers are pseudonymised before anyone does analysis. A Python model produces risk scores and explanations, and Power BI shows aggregated views with row-level security so a manager sees only their own team.</p><p>The model is the smallest box in the diagram. The guardrails matter more, and they live in the pipeline rather than in a policy document. Analysts work on hashed IDs, no chart or table shows fewer than ten people, access is by role, and no score triggers an action automatically. A human always decides.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*53zVMLOuAoRSaPqh1_50AA.png" /></figure><p>Build It: An Attrition Risk Model in Python</p><p>Here is a working example. The data is synthetic: 1,470 fictional employees that I generated with a few built-in relationships, such as overtime and low engagement raising the chance of leaving. It has the usual fields: department, tenure, overtime, salary relative to the pay band for the role, months since last promotion, manager changes in the last two years and an engagement score. Roughly 17 percent of them leave, which is a realistic attrition rate.</p><p>The key decision comes before any modelling. Gender and age sit in the dataset so I can audit the results, but they are not allowed into the model. Here is the core of it:</p><pre>X = pd.get_dummies(<br>    df.drop(columns=[&quot;left&quot;, &quot;gender&quot;, &quot;age&quot;]),<br>    columns=[&quot;dept&quot;], drop_first=True)<br>y = df[&quot;left&quot;]<br><br>X_train, X_test, y_train, y_test, df_train, df_test = train_test_split(<br>    X, y, df, test_size=0.3, random_state=7, stratify=y)<br><br>model = GradientBoostingClassifier(<br>    n_estimators=150, max_depth=3, learning_rate=0.05, random_state=7)<br>model.fit(X_train, y_train)<br>risk = model.predict_proba(X_test)[:, 1]<br>print(roc_auc_score(y_test, risk))</pre><p>The model scored an AUC of 0.66. That is better than guessing and nowhere near a crystal ball, and I would rather show you that number than a flattering one. A score that low would never justify a decision about an individual. What it can do is rank where to look first.</p><p>The more useful measure is lift. If you take the 10 percent of people the model ranks as highest risk, 31.8 percent of them actually left, against a base rate of 17.2 percent. That is about 1.85 times better than picking at random. The top 20 percent captured 35.5 percent of all leavers. For an HR team with limited time for stay conversations, that is a genuinely useful ordering, as long as it is used as a prompt to talk and not a verdict.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FWHDATADOIj_1fmelCOYGg.png" /></figure><p>Then I asked what was driving the scores, using permutation importance, which shuffles one column at a time and measures how much the model gets worse. Engagement came first, then overtime, then pay relative to the band. Months since promotion and department barely registered. Notice that the top drivers are all things a company can change. That is where the real value is. The conversation worth having is not who is going to leave but why overtime is so heavy in one team.</p><h3>The Fairness Check Most Projects Skip</h3><p>Before I would show this to anyone, I ran a check on the people it flags. I labelled the top 20 percent as high-risk and compared how often women and men landed in that group. The usual rule of thumb, borrowed from US employment practice, is the four-fifths rule: the lower group&#39;s rate should be at least 80 percent of the higher group&#39;s. The code is short:</p><pre>flag = np.zeros(len(risk), dtype=bool)<br>flag[np.argsort(-risk)[: int(len(risk) * 0.2)]] = True<br><br>rates = (pd.DataFrame({&quot;gender&quot;: df_test[&quot;gender&quot;].values, &quot;flag&quot;: flag})<br>           .groupby(&quot;gender&quot;)[&quot;flag&quot;].mean())<br>print(rates)<br>print(&quot;ratio:&quot;, rates.min() / rates.max())</pre><p>The result: 21.5 percent of women and 18.8 percent of men were flagged, a ratio of 0.88, above the 0.80 line. I need to be honest about something, though. My first run of this code scored 0.76. While writing this section I noticed that I had accidentally left gender and age in the list of model features. Removing them moved the ratio to 0.88.</p><p>I would not claim that the gender column alone explains the difference. A test set of 441 people is small, and a ratio like this moves around with the random split. But it is exactly the kind of mistake that is easy to make and invisible unless you test for it, which is why the check goes on the launch checklist. Even with protected attributes removed, other fields can act as proxies, so the test has to run on the output, not just the inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8CXNC2ndIvyrdvqkZ00gUQ.png" /></figure><p>Putting It in Power BI</p><p>The model output goes into the warehouse as a table of team-level risk, never individual names. In Power BI, I keep the measures simple and the visuals aggregated. These are the ones I start with:</p><pre>Leavers = CALCULATE(COUNTROWS(Employees), Employees[Left] = 1)<br><br>Average Headcount =<br>AVERAGEX(VALUES(&#39;Date&#39;[Month]),<br>    CALCULATE(COUNTROWS(Employees), Employees[Active] = 1))<br><br>Attrition Rate = DIVIDE([Leavers], [Average Headcount])<br><br>Safe Attrition Rate = IF([Average Headcount] &lt; 10, BLANK(), [Attrition Rate])</pre><p>That last measure is the one I care about most. It returns blank for any team with fewer than ten people, so a manager can never work out who a number refers to. Add row-level security so managers see only their own part of the organisation, and the dashboard becomes something people can trust instead of something they fear.</p><h3>Using Language Models Carefully</h3><p>The other place AI works well in HR is text. Free-text survey comments are rich and almost nobody has time to read them. A language model can group a few thousand comments into themes, summarise each theme and pull a few representative quotes. I use it that way, with three rules. Strip names and identifying details before the text goes anywhere. Use a provider and contract setup that your data protection team has approved. And treat the output as a starting point for a human to read, never as a conclusion.</p><p>What I do not do is ask a model to score or rank individual employees from their comments or emails. It is tempting, and it is exactly the kind of use that has caused problems elsewhere.</p><h3>A Checklist Before Anything Goes Live</h3><p>These are the questions I want answered before an HR model touches a real decision. What decision will this support, and what is it explicitly not allowed to be used for? Which fields are in the model, and have I tested the output for fairness across groups? How accurate is it per group, not just overall? Who can see individual-level scores, and why? How do employees find out about it, and is there a way to question a result? Who reviews it again in six months, because workforces change and models drift?</p><p>Notice that none of these are technical questions. The technical work is the easier part.</p><h3>Where to Start</h3><p>If you are leading an HR analytics effort, I would start well below machine learning. Get one clean employee table that joins the HR system with payroll and survey data. Build the descriptive and diagnostic dashboards, attrition by manager, tenure, pay position and workload, and you will probably find the next quarter&#39;s actions without a model. Add prediction only when someone can name the decision it supports and the intervention that follows.</p><p>The best HR analytics projects I have seen were not the cleverest. They helped a manager have a better conversation with a tired team. That is a good bar for any model: if it does not make a human conversation better, it is probably not worth building.</p><p>If you are working on something similar and want a second opinion, I write about Power BI, SQL and practical analytics, and you can find more of my work at <a href="https://iamgaurav.netlify.app/">iamgaurav.netlify.app</a>.</p><h3>Sources</h3><p>The attrition model and its numbers come from a synthetic dataset I generated; everything else below is public.</p><ul><li>Amazon scraps a secret AI recruiting tool that showed bias against women (CNBC, Oct 2018) https://www.cnbc.com/2018/10/10/amazon-scraps-a-secret-ai-recruiting-tool-that-showed-bias-against-women.html</li><li>IBM AI can predict with 95% accuracy which employees will quit (CNBC, Apr 2019) https://www.cnbc.com/2019/04/03/ibm-ai-can-predict-with-95-percent-accuracy-which-employees-will-quit.html</li><li>Google Studies Management and Uncovers the Fundamentals (Josh Bersin) https://joshbersin.com/2011/03/google-studies-management-and-uncovers-the-basics/</li><li>HireVue drops facial monitoring amid A.I. algorithm audit (Fortune, Jan 2021) https://fortune.com/2021/01/19/hirevue-drops-facial-monitoring-amid-a-i-algorithm-audit</li><li>HireVue, Facing FTC Complaint From EPIC, Halts Use of Facial Recognition (EPIC) https://epic.org/hirevue-facing-ftc-complaint-from-epic-halts-use-of-facial-recognition/</li><li>Critical audit of NYC AI hiring law signals increased risk for employers (DLA Piper, Jan 2026) https://www.dlapiper.com/en/insights/publications/2026/01/critical-audit-of-nyc-ai-hiring-law-signals-increased-risk-for-employers</li><li>AI bias lawsuit against Workday reaches next stage as court grants conditional certification of ADEA claim (Proskauer) https://www.proskauer.com/blog/ai-bias-lawsuit-against-workday-reaches-next-stage-as-court-grants-conditional-certification-of-adea-claim</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0be195638ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Business Reporting: Stop Rebuilding the Same Report Every Monday]]></title>
            <link>https://gauravkrvish.medium.com/automated-business-reporting-stop-rebuilding-the-same-report-every-monday-ebe42cc23c61?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebe42cc23c61</guid>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 04:58:39 GMT</pubDate>
            <atom:updated>2026-11-09T05:01:54.094Z</atom:updated>
            <content:encoded><![CDATA[<p>Every Monday at 7:30 a.m., somewhere in a finance team, someone exports a file from the ERP, opens last week&#39;s Excel workbook, and starts fixing the VLOOKUPs that broke over the weekend. By 10 the PDF goes out to fourteen people. By 10:05 someone replies that the returns number looks off.</p><p>I have been that someone, and I have also been the person who gets called in to replace that someone. The fix is almost never a fancier dashboard. It is a pipeline: data lands, gets checked, gets refreshed, and arrives in the inbox on its own, at the same hour, every week.</p><p>This article walks through how I build that pipeline with SQL, Power BI and a little Python. It covers what is worth automating, the checks that stop a wrong number from reaching your CFO, and the mistakes that cost me a few uncomfortable Monday mornings along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*drrCZ8zGDJJssN6SS8vQNg.png" /></figure><h3>Why Manual Reporting Costs More Than It Looks</h3><p>The hours are the part everyone notices. A weekly report that takes three hours to build is 150 hours a year, close to a month of one person&#39;s working time, spent on copying numbers from one place to another. Nobody budgets for that, so nobody sees it.</p><p>The bigger cost is trust. Manual reports break in quiet ways: a filter left on from last week, a pasted range that stops one row short, a lookup that returns the wrong match because someone sorted a column. Each one is small. But the first time a director catches a wrong number in a meeting, every later number gets a second look, and the report stops being the thing people decide from.</p><p>There is a third cost that is easy to miss. When the report depends on one person&#39;s Monday morning, it also depends on that person not being sick, on leave, or buried in something urgent. I have seen a company go two weeks without its sales report because the one analyst who knew the macro was out.</p><h3>What to Automate, and What to Leave Alone</h3><p>Not every report deserves a pipeline. My rule of thumb is simple: if a report is produced on a schedule, read by the same people, and built from the same sources each time, automate it. Weekly sales, monthly finance packs, daily operations snapshots, inventory and aging reports all fit that description.</p><p>What I leave manual is anything that needs judgment in the middle. A board pack with commentary, a one-off analysis for an acquisition, a report where the definitions are still being argued about. Automating those too early just locks in a definition nobody has agreed on yet, and you end up automating the argument.</p><p>A useful test before you start: can you write down, in plain sentences, where every number on the report comes from? If you cannot, the first job is not automation. It is getting that definition written down and signed off, because a pipeline will faithfully repeat whatever ambiguity you feed it.</p><h3>The Pipeline at a Glance</h3><p>Every automated report I build follows the same five stages. The tools change from client to client, but the shape does not. Data comes from a source system, gets shaped into one clean model, gets checked, feeds a refreshed report, and is delivered to the people who need it.</p><p>Most teams automate stages 4 and 5, the refresh and the email, and skip stage 3 entirely. That is backwards. Scheduling a refresh is the easy part. Making sure the numbers are right before the email goes out is the part that decides whether anyone keeps trusting the report.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TFCdrnft8HPPlxP6kV0fjw.png" /></figure><h3>Step 1: Build One Clean Reporting Layer</h3><p>Before anything is scheduled, the report needs a single place to read from. I do not point Power BI at raw tables. I create a reporting schema in the warehouse and put one view per report in it. The view does the joins, applies the business rules, and returns exactly the columns the report needs, nothing more.</p><p>The benefit shows up months later. When finance changes how returns are counted, you change one view. The report, the PDF and the email all follow, and nobody has to remember which Excel file had the old formula. Here is a trimmed version of the weekly sales view I use as a starting point:</p><pre>CREATE OR REPLACE VIEW reporting.v_weekly_sales AS<br>SELECT<br>    d.week_start,<br>    r.region_name,<br>    SUM(f.net_amount)          AS revenue,<br>    COUNT(DISTINCT f.order_id) AS orders,<br>    SUM(f.returned_amount) / NULLIF(SUM(f.gross_amount), 0) AS return_rate<br>FROM core.fact_sales f<br>JOIN core.dim_date   d ON d.date_key   = f.date_key<br>JOIN core.dim_region r ON r.region_key = f.region_key<br>WHERE f.is_test_order = FALSE<br>GROUP BY d.week_start, r.region_name;</pre><p>A few lines in that view do more work than they look. The is_test_order filter keeps QA orders out of the totals. NULLIF stops a division by zero from breaking the refresh on a quiet week. And because the grain is one row per week per region, the report never has to aggregate anything surprising on its own.</p><h3>Step 2: Put the Data Checks Before the Report</h3><p>This is the step most automation projects skip, and it is the one I would never skip again. A refresh that finishes successfully only tells you that the query ran. It says nothing about whether the data underneath was complete.</p><p>I learned this the slow way. An upstream job failed halfway one night, the warehouse had about 8 percent fewer rows than usual, and the report refreshed without a single error. It went out showing a revenue dip that did not exist. The client found out because their regional manager phoned the CFO to ask what had gone wrong in the south.</p><p>Now every report gets a small set of checks that run before the refresh. Mine are deliberately boring: how old is the newest row, are any key columns empty, and does the row count match the staging layer. If any of them fail, the pipeline stops and sends a message instead of a report.</p><pre>CHECKS = {<br>    &quot;hours_since_load&quot;: (&quot;SELECT DATEDIFF(&#39;hour&#39;, MAX(loaded_at), CURRENT_TIMESTAMP) FROM core.fact_sales&quot;,<br>                         lambda v: v &lt;= 24),<br>    &quot;null_region_keys&quot;: (&quot;SELECT COUNT(*) FROM core.fact_sales WHERE region_key IS NULL&quot;,<br>                         lambda v: v == 0),<br>    &quot;rows_vs_staging&quot;: (&quot;SELECT ABS((SELECT COUNT(*) FROM core.fact_sales) - (SELECT COUNT(*) FROM staging.sales_raw))&quot;,<br>                         lambda v: v == 0),<br>}<br><br>def run_checks(cur):<br>    failures = []<br>    for name, (sql, is_ok) in CHECKS.items():<br>        value = cur.execute(sql).fetchone()[0]<br>        if not is_ok(value):<br>            failures.append(f&quot;{name} = {value}&quot;)<br>    return failures</pre><p>The thresholds are the part you tune. 24 hours is right for a nightly load, while a report built on an hourly feed would use 2. When a check fails, the function returns a list of what went wrong, and the next stage decides what to do with it. More on that in step 5.</p><h3>Step 3: Schedule the Refresh in Power BI</h3><p>With the checks passing, the refresh itself is the easy part. In the Power BI service, open the semantic model settings and set a scheduled refresh. Pick a time after your warehouse job normally finishes, not at the same moment. I leave a 60 to 90 minute buffer, because pipelines run late far more often than they run early.</p><p>Two settings are worth getting right. First, turn on failure notifications to yourself and one other person, so a failed refresh does not wait for Monday&#39;s email to be noticed. Second, connect through a service account or a shared cloud connection, never your own login. I once got a weekend call because a report quietly depended on a colleague&#39;s password, and it expired.</p><p>One limit to know about: a Pro license allows 8 scheduled refreshes a day per model, and Premium or Fabric capacity allows 48. For a daily or weekly report that is plenty. If you want the refresh to run only after the checks pass, skip the schedule and call the refresh from the same script through the REST API instead.</p><h3>Step 4: Turn the Report into a PDF and Send It</h3><p>You have two routes here, and the right one depends on your license. The built-in route is a Power BI subscription. It emails a snapshot of the report on a schedule with a link back to the live version. It takes five minutes to set up and for many teams it is all they need.</p><p>The second route is the export API, which gives you a proper PDF you can attach to your own email, name the way finance likes, and send only when the checks have passed. The catch is that the report has to sit in a workspace on Premium, Premium Per User or Fabric capacity. If it does not, use a subscription and do not fight it.</p><p>The export is asynchronous. You start a job, poll until it finishes, then download the file. This is the core of the script I reuse, with authentication and error handling trimmed out for space:</p><pre>BASE = &quot;https://api.powerbi.com/v1.0/myorg/groups/{ws}/reports/{rep}&quot;<br><br>def export_pdf(token, ws, rep):<br>    h = {&quot;Authorization&quot;: f&quot;Bearer {token}&quot;}<br>    url = BASE.format(ws=ws, rep=rep)<br>    job = requests.post(f&quot;{url}/ExportTo&quot;, headers=h, json={&quot;format&quot;: &quot;PDF&quot;}).json()<br><br>    while True:<br>        s = requests.get(f&quot;{url}/exports/{job[&#39;id&#39;]}&quot;, headers=h).json()<br>        if s[&quot;status&quot;] in (&quot;Succeeded&quot;, &quot;Failed&quot;):<br>            break<br>        time.sleep(15)<br><br>    if s[&quot;status&quot;] != &quot;Succeeded&quot;:<br>        raise RuntimeError(&quot;Power BI export failed&quot;)<br>    return requests.get(f&quot;{url}/exports/{job[&#39;id&#39;]}/file&quot;, headers=h).content</pre><p>Attaching those bytes to an email is ordinary SMTP or your mail provider&#39;s API, so I will not paste it here. The part worth copying is the order of operations: check, refresh, export, send. If any earlier step fails, nothing goes out.</p><h3>Step 5: Tell Someone When It Breaks</h3><p>Automation without alerting is just a quieter way to fail. The moment a check trips or an export errors out, a human has to hear about it, in a place they actually look. For most teams that is a Teams or Slack channel, not an inbox that fills up with automated mail.</p><p>A good alert says three things: which report is held, which check failed, and what the value was. &quot;Weekly sales held: rows_vs_staging = 3778&quot; is something an engineer can act on in a minute. &quot;Pipeline failed&quot; is not. Here is the whole notifier and the decision around it:</p><pre>def notify_teams(webhook_url, report, failures):<br>    lines = &quot;\n&quot;.join(f&quot;- {f}&quot; for f in failures)<br>    text = f&quot;{report} held. Checks failed:\n{lines}&quot;<br>    requests.post(webhook_url, json={&quot;text&quot;: text}, timeout=10)<br><br>failures = run_checks(cur)<br>if failures:<br>    notify_teams(WEBHOOK_URL, &quot;Weekly sales&quot;, failures)<br>else:<br>    pdf = export_pdf(token, WORKSPACE_ID, REPORT_ID)<br>    send_email(RECIPIENTS, &quot;Weekly Sales Report&quot;, pdf)</pre><p>One practical note: Microsoft has been moving Teams from the old connector webhooks to Workflows, so check which kind of webhook URL your tenant supports. The code barely changes, and Slack, email or a phone push work the same way.</p><h3>What Actually Changed</h3><p>On the first project where I built this end to end, a weekly sales report went from about three hours of one analyst&#39;s Monday to roughly twenty minutes of reading exceptions. That is the number people quote, but it is not the one I care about most.</p><p>What changed more was behavior. The report started arriving at 8:00 every Monday whether or not anyone remembered it. Within a few weeks the regional managers stopped asking whether the numbers were final and started asking what the numbers meant. That is what a report is for. Nobody argues with a report they trust.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oa0ETPe46UQN0wVYTDGzsA.png" /></figure><h3>Mistakes I Made So You Can Skip Them</h3><p>I automated before agreeing on definitions. My first version of the returns metric counted a return in the week it was processed, while finance counted it in the week of the original sale. The two reports disagreed by 0.6 points on the first automated run. Automation made the argument visible, which was useful, but settling it first would have been cheaper.</p><p>I edited the live view. Changing the reporting view in production while the 8:00 email was scheduled is exactly as risky as it sounds. Now every change is tested in a dev schema first, and the view lives in version control like any other code.</p><p>I trusted silence. When the pipeline broke early on, the email simply did not arrive, and nobody noticed for a week because an absence does not trigger anything. I now post a short heartbeat message to the team channel every time a run completes, so a missing message becomes the alert.</p><p>I left the report without an owner. Every automated report needs one named person who gets the alerts and decides what to do with them. A pipeline that belongs to everyone fails quietly until it embarrasses someone.</p><h3>Where to Start This Week</h3><p>Pick one report. Choose the one that is weekly, annoying to build, and read by the same few people. Then work in this order: write down where each number comes from, move the logic into a view, add three checks, schedule the refresh, and automate the delivery last.</p><p>If you only have an afternoon, stop after the checks and the scheduled refresh. Even without the email, you will have removed the riskiest and most manual part of the job. And resist the urge to build a perfect framework first. My first working version was one SQL view and one Python file, and it ran for months before it deserved anything fancier.</p><p>The goal was never to remove the analyst from reporting. It is to move them from assembling the report to understanding it, which is the part they were hired for in the first place.</p><p>If you are stuck in a Monday-morning reporting loop like the one at the top of this article, I write about Power BI, SQL and practical analytics, and you can find more of my work at <a href="https://iamgaurav.netlify.app/">iamgaurav.netlify.app</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebe42cc23c61" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build an AI-Powered Business Intelligence Dashboard]]></title>
            <link>https://gauravkrvish.medium.com/build-an-ai-powered-business-intelligence-dashboard-2cad2db4bc5c?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cad2db4bc5c</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 04:37:41 GMT</pubDate>
            <atom:updated>2026-11-09T04:37:41.079Z</atom:updated>
            <content:encoded><![CDATA[<p>Last quarter, a client called me in a panic. Their Power BI dashboard was showing revenue numbers that made no sense — a 40% spike on a Tuesday, followed by a crash on Wednesday. The business team was convinced something was broken. The data team was convinced the business team was reading it wrong.</p><p>Both were right. The anomaly was real. But nobody had built a system to explain it automatically.</p><p>That conversation is why I spent the next three months building an AI-powered BI dashboard that doesn’t just show data — it understands it, queries it, and tells you what matters. This is the technical story of how I built it and what I learned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dCHnCgPcDfriLYLGLc_hVg.png" /></figure><h3>What “AI-Powered” Actually Means in BI</h3><p>Before I walk you through the build, let me be direct about something: most dashboards labeled “AI-powered” are just dashboards with a chatbot slapped on top. That’s not what we’re building here.</p><p>A genuinely AI-powered BI system does four things traditional dashboards can’t:</p><p>It answers questions in plain English, not just pre-built charts. When a VP asks “why did northeast revenue drop 15% last week?”, the system queries the data and gives an actual answer — not a filtered chart that the analyst has to interpret.</p><p>It detects anomalies automatically without manual threshold-setting. Instead of someone deciding “flag anything above 20% variance”, the system learns normal patterns and flags deviations from those patterns.</p><p>It generates narrative explanations in natural language. Numbers alone don’t drive decisions. Context does. The system writes the paragraph that explains what the numbers mean.</p><p>It integrates with your existing data infrastructure. This isn’t a standalone tool — it lives inside Power BI where your business users already work.</p><h3>The Stack</h3><p>Here&#39;s what I chose and why each component earned its place.</p><p>Power BI as the presentation layer. Your users already live here. Fighting that is a losing battle. The goal is to make Power BI smarter, not replace it.</p><p>Snowflake as the data warehouse. The SQL dialect is clean, performance on large analytical queries is excellent, and the Python connector is straightforward.</p><p>FastAPI as the middleware layer. A Python microservice that sits between Power BI and the AI components, so the AI logic never pollutes your data layer.</p><p>LangChain plus GPT-4o for natural language queries. LangChain&#39;s SQL agent handles the translation from English to SQL, and GPT-4o&#39;s reasoning covers the ambiguous cases where a question could mean more than one thing.</p><p>Facebook Prophet for anomaly detection. Time series forecasting with clean uncertainty intervals. I looked at several options and Prophet&#39;s interpretability won. When you tell a client their Tuesday spike was anomalous, you need to be able to explain why the system thinks so.</p><h3>The Architecture</h3><p>The core design decision: Power BI stays dumb.</p><p>Every AI call goes through the FastAPI layer. Power BI talks to FastAPI, and FastAPI talks to Snowflake and the AI components. This matters for three reasons. It keeps your Power BI reports portable, so you can rebuild the frontend without touching the AI logic. It lets you version and test the AI components independently. And it stops sensitive data from flowing through the models unnecessarily, because FastAPI controls exactly what gets sent to GPT-4o.</p><p>The request flow looks like this. A Power BI Python visual sends a question or metric to FastAPI. FastAPI routes it to either the SQL agent (for ad-hoc questions) or the Prophet analyzer (for anomaly detection). Results come back as JSON, and Power BI renders them as text, charts, or both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MNsawuCfEJ9K2NnGn1FM6Q.png" /></figure><h3>Step 1: The Data Foundation</h3><p>Good AI output needs clean data input. I created a consistent schema layer in Snowflake that the SQL agent could navigate reliably.</p><p>The key table is REPORTING_METADATA. It stores column descriptions, business-friendly names, and example values for every table the AI is allowed to query. This matters enormously. When the SQL agent knows that CUST_REV_AMT means &quot;customer revenue in USD, net of refunds&quot;, it writes much better SQL than when it&#39;s guessing.</p><p>This is the schema pattern that worked well:</p><pre>CREATE TABLE REPORTING_METADATA (<br>  table_name VARCHAR,<br>  column_name VARCHAR,<br>  business_name VARCHAR,<br>  description VARCHAR,<br>  example_values VARCHAR,<br>  is_queryable BOOLEAN<br>);</pre><h3>Step 2: Natural Language Query Layer</h3><p>This is where LangChain and GPT-4o come in. The goal is simple: take a plain English question from a Power BI user and return accurate data.</p><pre>from langchain_community.utilities import SQLDatabase<br>from langchain_community.agent_toolkits import create_sql_agent<br>from langchain_openai import ChatOpenAI<br><br>llm = ChatOpenAI(model=&quot;gpt-4o&quot;, temperature=0)<br>db = SQLDatabase.from_uri(snowflake_uri, include_tables=REPORTING_TABLES)<br>agent = create_sql_agent(llm=llm, db=db, verbose=True, agent_type=&quot;openai-tools&quot;)<br><br>def query_data(natural_language_question: str) -&gt; dict:<br>    result = agent.invoke({&quot;input&quot;: natural_language_question})<br>    return {&quot;answer&quot;: result[&quot;output&quot;], &quot;sql&quot;: extract_sql(result)}</pre><p>Temperature is set to 0 for SQL generation. This is deliberate. You want deterministic query construction, not creative interpretation.</p><p>The FastAPI endpoint that Power BI calls:</p><pre>@app.post(&quot;/query&quot;)<br>async def query_endpoint(request: QueryRequest):<br>    result = query_data(request.question)<br>    return {&quot;data&quot;: result[&quot;answer&quot;], &quot;sql_used&quot;: result[&quot;sql&quot;]}</pre><p>One important detail: I always return the SQL used alongside the answer. This lets analysts audit the queries and catch cases where the model misread the question. It&#39;s also a trust-building feature. When users can see the query, they trust the answer more.</p><h3>Step 3: Anomaly Detection with Prophet</h3><p>For each key metric, Prophet builds a model of &quot;normal&quot; and flags anything outside the 95% confidence interval.</p><pre>def detect_anomalies(metric_df: pd.DataFrame, metric_name: str) -&gt; dict:<br>    model = Prophet(<br>        yearly_seasonality=True,<br>        weekly_seasonality=True,<br>        daily_seasonality=False,<br>        interval_width=0.95<br>    )<br>    model.fit(metric_df)<br>    future = model.make_future_dataframe(periods=0)<br>    forecast = model.predict(future)<br><br>    results = metric_df.merge(<br>        forecast[[&#39;ds&#39;, &#39;yhat&#39;, &#39;yhat_lower&#39;, &#39;yhat_upper&#39;]], on=&#39;ds&#39;<br>    )<br>    results[&#39;is_anomaly&#39;] = (<br>        (results[&#39;y&#39;] &gt; results[&#39;yhat_upper&#39;]) |<br>        (results[&#39;y&#39;] &lt; results[&#39;yhat_lower&#39;])<br>    )<br>    results[&#39;deviation_pct&#39;] = (<br>        (results[&#39;y&#39;] - results[&#39;yhat&#39;]) / results[&#39;yhat&#39;] * 100<br>    ).round(1)<br><br>    anomalies = results[results[&#39;is_anomaly&#39;]].to_dict(&#39;records&#39;)<br>    return {<br>        &quot;metric&quot;: metric_name,<br>        &quot;anomaly_count&quot;: len(anomalies),<br>        &quot;anomalies&quot;: anomalies<br>    }</pre><p>Why Prophet over a simple threshold? Because thresholds break in seasonal businesses. A 20% revenue increase in December is normal. The same spike in February is worth investigating. Prophet handles this automatically by modeling weekly and yearly seasonality.</p><h3>Step 4: Narratives via GPT-4o</h3><p>Raw anomaly data isn&#39;t actionable. This component turns the JSON output into a readable explanation. This is the prompt template I use:</p><pre>You are a business analyst explaining data to executives.<br>Given these anomalies in {metric_name}: {anomaly_data}<br>Write a 2-3 sentence explanation of what happened and why it might matter.<br>Be specific about dates and magnitudes. Avoid jargon.<br>Maximum 80 words.</pre><p>The 80-word limit is intentional. Long AI explanations get ignored, short and specific ones get read. I use temperature 0.3 here: slightly creative, mostly factual.</p><h3>Step 5: The Power BI Python Visual</h3><p>This is what ties everything together. Power BI&#39;s Python visual sends requests to FastAPI and renders the response.</p><pre>import requests<br>import matplotlib.pyplot as plt<br><br>endpoint = &quot;https://your-fastapi-service.com&quot;<br><br>def get_anomaly_narrative(metric_name, days_back=30):<br>    response = requests.post(<br>        f&quot;{endpoint}/analyze&quot;,<br>        json={&quot;metric&quot;: metric_name, &quot;days&quot;: days_back}<br>    )<br>    return response.json()<br><br>result = get_anomaly_narrative(&quot;daily_revenue&quot;)<br><br>fig, ax = plt.subplots(figsize=(10, 2))<br>ax.text(0.05, 0.5, result[&#39;narrative&#39;],<br>        transform=ax.transAxes,<br>        fontsize=11, verticalalignment=&#39;center&#39;,<br>        wrap=True)<br>ax.axis(&#39;off&#39;)<br>plt.tight_layout()<br>plt.show()</pre><p>The visual renders as a text box inside Power BI. When a user opens the revenue page, they see the usual charts plus a plain-language summary at the top explaining any anomalies from the past 30 days.</p><h3>What Actually Impressed the Client</h3><p>Three months in, the system flagged a 23% revenue dip on a Saturday and generated this narrative: &quot;Saturday revenue dropped 23% below expected levels (actual: $847K vs expected: $1.1M). This deviation falls outside the 95% confidence interval for weekend revenue. The dip coincides with a Cloudflare incident affecting checkout from 2-6 PM EST.&quot;</p><p>The last sentence came from connecting the anomaly detector to incident log data. That correlation took two hours to build. The client&#39;s reaction made those two hours worthwhile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nPg2DMEsjSUHaMCyn8qY5A.png" /></figure><h3>Lessons Learned</h3><p>The metadata layer is the highest-leverage investment. Spending a week writing good column descriptions improved query accuracy more than any prompt engineering trick.</p><p>Audit trails matter more than accuracy. Users will trust an occasionally wrong system that shows its work over a black box that&#39;s usually right.</p><p>Start with one metric, not all of them. I tried to instrument everything at once and it was a maintenance nightmare. Pick the metric that causes the most analyst pain and get that one right first.</p><p>The LangChain SQL agent struggles with multi-step reasoning. A question like &quot;which product category grew fastest last quarter among customers who joined this year&quot; needs several sub-queries. I ended up building a few specialized query functions for the most common complex questions instead of leaning on the agent.</p><h3>Where to Go From Here</h3><p>The system I&#39;ve described is functional but not finished. The natural next steps are these.</p><p>Streaming responses, so users see results as they&#39;re generated instead of waiting for the full API response.</p><p>Feedback loops, where analysts can flag incorrect queries and that data feeds back into the prompt.</p><p>Multi-modal outputs that return both a narrative and a recommended visualization type for the data.</p><p>Scheduled anomaly reports delivered to Slack or email, so nobody has to open Power BI to find out something broke.</p><p>The gap between a dashboard that shows data and a dashboard that understands data is significant engineering work. But it&#39;s the kind of work that actually changes how organizations make decisions.</p><p>That Tuesday spike my client panicked about? It turned out to be a flash sale the marketing team had forgotten to flag in the revenue data. The system caught it, explained it, and saved three hours of analyst investigation.</p><p>That&#39;s the goal. Not AI for AI&#39;s sake, but AI that reduces the distance between data and decisions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cad2db4bc5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Data Governance: Why Most AI Problems Are Really Data Problems]]></title>
            <link>https://gauravkrvish.medium.com/ai-data-governance-why-most-ai-problems-are-really-data-problems-661e8fcfdc86?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/661e8fcfdc86</guid>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-governance]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Sat, 07 Nov 2026 03:31:02 GMT</pubDate>
            <atom:updated>2026-11-07T03:31:02.236Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uP3IksoZUbMXZ4SxXSJbpg.png" /></figure><p>Most conversations about AI at work start with the model. Which one is smarter, which one is cheaper, which one writes better SQL. Very few start with the boring question: what data is this thing allowed to see, and is that data any good?</p><p>That boring question is where most AI projects actually go wrong.</p><p>Gartner surveyed data management leaders in 2024 and came back with two numbers worth printing out. Through 2026, they expect organisations to abandon <strong>60% of AI projects</strong> that aren’t backed by AI-ready data. And <strong>63%</strong> of organisations either don’t have, or aren’t sure they have, the right data management practices for AI. That’s not a model problem. That’s a governance problem.</p><p>I work with data every day, mostly reporting and dashboards, and lately more and more AI on top of that data. This post is the practical version of AI data governance: what it is, how it’s different from the governance most companies already have, the rules now arriving in India and Europe, and a plan you can start in 90 days without hiring a consulting firm.</p><h3>What “AI data governance” actually means</h3><p>Strip away the jargon and it’s four questions:</p><ol><li><strong>What data</strong> is the AI using (to train, to look things up, or in the prompt)?</li><li><strong>Who</strong> is allowed to see that data, and does the AI respect that?</li><li><strong>Is the data good enough</strong> to make decisions on: accurate, complete, recent and fair?</li><li><strong>Can you prove it</strong> afterwards: where an answer came from, who owns the data, what was logged?</li></ol><p>Traditional data governance already asks versions of these questions about your warehouse and your reports. AI doesn’t replace that work. It stretches it in directions most governance programmes weren’t built for.</p><h3>How it’s different from the governance you already have</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-p_bjmOQsVa3Calfgt93-w.png" /></figure><p>A few things change once AI is in the picture.</p><p><strong>The data isn’t just tables any more.</strong> Classic governance covers the warehouse: customer tables, sales facts, a data dictionary. AI assistants read PDFs, SharePoint folders, email threads, ticket histories and meeting transcripts. Most companies have never classified any of that. Nobody knows which folder has salary letters in it until an assistant finds them.</p><p><strong>Prompts and outputs are data too.</strong> When someone pastes a customer list into a chatbot to “clean it up”, that’s personal data leaving your systems. When the AI writes a summary that ends up in a board pack, that output is now a record someone will rely on. Both need rules.</p><p><strong>Access control has to work on the AI’s behalf.</strong> A report either opens for you or it doesn’t. An AI assistant that searches documents can quietly pull a paragraph from a file you were never meant to open, and hand it to you in a friendly sentence. If the AI’s permissions are broader than the user’s, you’ve built a very polite data leak.</p><p><strong>“Correct” isn’t enough.</strong> A sales total can be correct. A model trained on five years of hiring decisions can be perfectly accurate at repeating the bias in those decisions. AI governance has to ask whether data is representative and fair, not just whether it’s clean.</p><p><strong>Lineage gets longer.</strong> In BI, lineage goes from source system to dashboard. With AI it runs from source to training set or document index, to prompt, to model, to answer. When someone asks “where did the AI get that?”, you need an answer that isn’t “we’re not sure”.</p><h3>The six pillars</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u73qKcAo9tpk_Yt47em0Xg.png" /></figure><p>Every framework I’ve read slices this a little differently. These are the six that matter in practice.</p><p><strong>1. Quality.</strong> AI amplifies whatever you feed it. Duplicate customers, missing months, three definitions of “active user”: the model doesn’t know any of that is wrong. Basic checks (completeness, freshness, duplicates, agreed definitions) are still the highest-return work you can do.</p><p><strong>2. Access.</strong> The AI should see only what the person asking is allowed to see. That means respecting existing permissions in search and retrieval, not giving a service account access to everything “to make it work”.</p><p><strong>3. Privacy and consent.</strong> Personal data should be used only for the purpose it was collected for. If customers gave you their phone number to deliver an order, training a marketing model on it is a different purpose. Under India’s new data protection rules, that difference matters (more below).</p><p><strong>4. Lineage.</strong> For any important AI answer, you should be able to trace which documents or tables it came from and which version of the data was used.</p><p><strong>5. Bias and fairness.</strong> Before data is used for decisions about people (hiring, lending, pricing, performance), check how well different groups are represented and whether outcomes differ for reasons you can’t defend.</p><p><strong>6. Monitoring and accountability.</strong> Each dataset has a named owner. Prompts, sources and outputs are logged. Someone reviews it regularly. When something goes wrong, there’s a person whose job it is to fix it.</p><h3>A realistic example of how this goes wrong</h3><p><em>This is an illustrative scenario, not a specific company, but versions of it are common.</em></p><p>A mid-sized company builds an internal HR assistant. Employees can ask it about leave policy, benefits and travel rules. To make it useful, the team points it at the whole HR shared drive, because that’s where the policies live.</p><p>The shared drive also contains appraisal letters, salary revision sheets and a few exit-interview notes. Nobody tagged those as sensitive because, until now, only HR could open the folder.</p><p>A few weeks after launch, someone asks: “What was the average increment in the analytics team last year?” The assistant finds a salary sheet, summarises it helpfully, and answers.</p><p>Nothing was hacked. The model did exactly what it was built to do. The failure was governance: no classification of the documents, no check that the assistant’s access matched the user’s, and no log review that would have caught it early.</p><p>The fix isn’t a better model. It’s the boring list: classify the folders, restrict what the assistant indexes, filter retrieval by the user’s permissions, log questions and sources, and have HR own the dataset.</p><h3>The rules are catching up</h3><p>Even if you don’t care about governance for its own sake, regulators now do.</p><h4>India: the DPDP Act and Rules</h4><p>India’s Digital Personal Data Protection Rules were notified in November 2025, with a phased rollout. The Consent Manager framework starts from 13 November 2026, and full compliance is due by <strong>13 May 2027</strong>. A few parts matter directly for AI work:</p><ul><li>Personal data must be used for the purpose the person consented to, and deleted once that purpose is served.</li><li>Personal data breaches must be reported, with a detailed report to the Data Protection Board within 72 hours.</li><li>Processing logs have to be kept for at least a year.</li><li>Penalties go up to ₹250 crore for failing to take reasonable security safeguards, and up to ₹200 crore for failing to report breaches or meet obligations around children’s data.</li></ul><p>If your AI assistant reads customer records, or your team pastes personal data into public AI tools, these rules apply to that.</p><h4>Europe: the EU AI Act, Article 10</h4><p>For high-risk AI systems, Article 10 of the EU AI Act is essentially a data governance checklist written into law. Training, validation and testing data must be “relevant, sufficiently representative, and to the best extent possible, free of errors and complete”. Providers have to document where data came from and why it was collected, how it was prepared, the assumptions behind it, and how they examined it for possible biases and data gaps.</p><p>The high-risk obligations have been pushed back to December 2027, but if you sell to European clients, they will start asking these questions well before then. Some already are.</p><h3>A 90-day plan to get started</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FUHI40hjF-RdOu_fYv3ztQ.png" /></figure><p>You don’t need a governance council and a 60-page policy to start. You need owners, visibility and a few rules people actually follow.</p><p><strong>Days 1 to 30: see what you have.</strong></p><ul><li>List every AI tool in use, including the ones people signed up for themselves. Ask teams directly; you’ll be surprised.</li><li>For each tool, write down what data it touches: databases, folders, uploaded files, prompts.</li><li>Name one owner for each important dataset. A person, not a department.</li></ul><p><strong>Days 31 to 60: close the big gaps.</strong></p><ul><li>Restrict what AI assistants can index and retrieve. Start by excluding HR, finance and legal folders until they’re classified.</li><li>Write a one-page AI acceptable-use policy: what can go into public AI tools, what never can, and who to ask.</li><li>Stop personal data going into prompts where it doesn’t need to. Mask or remove names, phone numbers and IDs before data reaches the model.</li></ul><p><strong>Days 61 to 90: make it routine.</strong></p><ul><li>Add quality checks to the data feeding AI, the same way you’d test data feeding a finance report.</li><li>For any AI that informs decisions about people, run a basic bias check across groups before it goes live.</li><li>Log prompts, sources and outputs for internal AI tools. Review a sample every month and report what you find to leadership.</li></ul><h3>A quick checklist</h3><p>If you can answer yes to most of these, you’re ahead of most companies:</p><ul><li>Do we know every AI tool our people use?</li><li>Does each important dataset have a named owner?</li><li>Can our AI assistants only retrieve what the user is allowed to see?</li><li>Do we have a written rule on what can’t go into public AI tools?</li><li>Can we trace an AI answer back to its source documents or tables?</li><li>Have we checked the data behind people-related AI for bias?</li><li>Are we logging AI usage, and does anyone look at the logs?</li><li>Do we know which of our AI uses involve personal data under DPDP or GDPR?</li></ul><h3>Where this leaves us</h3><p>The companies getting real value from AI right now aren’t the ones with the fanciest model. They’re the ones whose data was already reasonably clean, owned and controlled, so they could plug AI in without creating a new mess.</p><p>If your data isn’t there yet, that’s normal. Start with the inventory and the owners. Everything else gets easier once you know what you have and who’s responsible for it.</p><p>If you’re building AI on top of your company’s data, these might help too:</p><ul><li><a href="https://gauravkrvish.medium.com/text-to-sql-explained-with-practical-examples-and-where-it-quietly-goes-wrong-96599835f40a">Text-to-SQL Explained with Practical Examples</a></li><li><a href="https://gauravkrvish.medium.com/i-built-an-ai-data-analyst-from-natural-language-to-sql-to-business-insights-831539717fa3">I Built an AI Data Analyst: From Natural Language to SQL to Business Insights</a></li><li><a href="https://medium.com/p/219a0d959a4e">Agentic AI vs Generative AI: What’s the Difference, and When to Use Which</a></li><li><a href="https://gauravkrvish.medium.com/ai-in-hr-reporting-what-it-should-automate-and-what-it-shouldnt-e10ab446de5a">AI in HR Reporting: What It Should Automate, and What It Shouldn’t</a></li></ul><p>I help teams with data, dashboards and practical AI. You can find my work at <a href="https://iamgaurav.netlify.app/">iamgaurav.netlify.app</a>.</p><h4>Sources</h4><ul><li>Gartner survey on AI-ready data, reported February 2025, via <a href="https://www.freevacy.com/news/gartner/gartner-reveals-60-of-ai-projects-with-data-issues-will-fail/6155">Freevacy</a></li><li>Fisher Phillips, <a href="https://www.fisherphillips.com/en/insights/insights/indias-new-data-privacy-rules-are-here">India’s New Data Privacy Rules Are Here</a></li><li>European Commission AI Act Service Desk, <a href="https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-10">Article 10: Data and Data Governance</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=661e8fcfdc86" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NLP, LLMs and Network Analytics for Social Media: A Real Case Study on 5,328 Election Tweets]]></title>
            <link>https://gauravkrvish.medium.com/nlp-llms-and-network-analytics-for-social-media-a-real-case-study-on-5-328-election-tweets-b32a1cbc9760?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/b32a1cbc9760</guid>
            <category><![CDATA[network-analysis]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[social-media-analytics]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 03:31:01 GMT</pubDate>
            <atom:updated>2026-11-06T03:31:01.786Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TXfi2IAR_sYWAswKxtmA9Q.png" /></figure><blockquote><strong><em>Disclaimer:</em></strong><em> This article is for academic and educational purposes only. It uses a small, publicly collected sample of tweets about the 2020 US presidential election to demonstrate text and network analysis methods. It is not political commentary, does not support or oppose any candidate or party, and the results do not measure public opinion or predict any election outcome. Individual users are not named; only public figures, official accounts and news organisations appear in the network results.</em></blockquote><p>Most social media dashboards I’ve seen answer one question: is the sentiment going up or down? That’s useful, but it leaves out most of what’s actually going on. Who is talking to whom? Which accounts sit in the middle of the conversation? Is a spike in negative sentiment one angry crowd or several different groups reacting to different things?</p><p>Answering those questions needs three things working together: <strong>NLP</strong> to read the text, <strong>LLMs</strong> to handle what simpler NLP can’t (sarcasm, insults, who a tweet is actually aimed at), and <strong>network analytics</strong> to map the structure of the conversation.</p><p>In this post I go through all three on a real dataset: 5,328 tweets I collected in 2020 for an election sentiment project that’s <a href="https://github.com/GAURAV19999/USA-Election-Sentiment-Analysis">public on my GitHub</a>. I re-ran everything for this article with current tools and kept every number from the actual run, including the results that made my original analysis look worse.</p><h3>The data, and the first surprise</h3><p>The dataset has two files: 2,788 tweets from a search around Donald Trump’s account and 2,540 from a search around Joe Biden’s. Just two columns, user and text. 4,964 different accounts, no retweets, no exact duplicates.</p><p>My original README says the tweets were collected in the week before the 3 November 2020 election. When I reread them for this post, that turned out to be wrong. The most common subject is a suggestion to delay the election, which Trump floated on 30 July 2020. Other tweets react to his “suburban lifestyle dream” post from 29 July and a government loan to Kodak announced on 28 July. The data is from the end of July 2020, three months before the vote.</p><p>Lesson one, before any model runs: <strong>check what period your data actually covers</strong>. A sentiment chart labelled “election week” would have been comparing the wrong news cycle.</p><p>The second thing I noticed: almost every tweet is a <strong>reply</strong>. All 2,540 Biden tweets mention @JoeBiden, and most Trump tweets mention @realDonaldTrump. That matters a lot for the network analysis later.</p><h3>The pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g43ShO2ZS41_FElg-miFbg.png" /></figure><p>This is the same general shape I’ve used for media monitoring and political social media work, just smaller:</p><ol><li><strong>Collect</strong> posts from an API or a monitoring tool.</li><li><strong>Clean</strong>: strip links, decode HTML entities, pull out @mentions and #hashtags into their own columns.</li><li><strong>NLP</strong>: sentiment, topics, entities.</li><li><strong>LLM layer</strong>: the cases where the cheap methods are unreliable.</li><li><strong>Network</strong>: build graphs from mentions and hashtags, then find central accounts and communities.</li><li><strong>Report</strong>: a dashboard (Power BI in my case) that combines what is said with who is saying it.</li></ol><pre>import pandas as pd<br>t = pd.read_csv(&quot;Trumpall2.csv&quot;); t[&quot;file&quot;] = &quot;Trump&quot;<br>b = pd.read_csv(&quot;Bidenall2.csv&quot;); b[&quot;file&quot;] = &quot;Biden&quot;<br>d = pd.concat([t, b], ignore_index=True)</pre><pre>d[&quot;mentions&quot;] = d.text.str.findall(r&quot;@(\w+)&quot;)<br>d[&quot;hashtags&quot;] = d.text.str.lower().str.findall(r&quot;#(\w+)&quot;)<br>d[&quot;clean&quot;] = (d.text.str.replace(r&quot;https?://\S+&quot;, &quot;&quot;, regex=True)<br>                    .str.replace(&quot;&amp;amp;&quot;, &quot;&amp;&quot;).str.strip())</pre><h3>Part 1: Sentiment, and why two standard tools disagree</h3><p>The two most common starting points for sentiment in Python are <strong>VADER</strong>, a rule-based lexicon built for social media, and <strong>TextBlob</strong>, a general-purpose lexicon. My original notebook used TextBlob. I ran both on the same cleaned text:</p><pre>from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer<br>from textblob import TextBlob</pre><pre>va = SentimentIntensityAnalyzer()<br>d[&quot;vader&quot;] = d.clean.map(lambda s: va.polarity_scores(s)[&quot;compound&quot;])<br>d[&quot;tb&quot;]    = d.clean.map(lambda s: TextBlob(s).sentiment.polarity)</pre><pre>def label(x, th):<br>    return &quot;positive&quot; if x &gt; th else &quot;negative&quot; if x &lt; -th else &quot;neutral&quot;<br>d[&quot;vader_lab&quot;] = d.vader.map(lambda x: label(x, 0.05))   # VADER&#39;s usual cut-off<br>d[&quot;tb_lab&quot;]    = d.tb.map(lambda x: label(x, 0))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vKqmiBdY2I8Waj14M9ERDQ.png" /></figure><p>The results tell two different stories:</p><ul><li><strong>VADER:</strong> 41.6% of the Trump-search tweets are negative against 24.6% positive. Biden-search tweets are 25.6% negative and 29.6% positive.</li><li><strong>TextBlob:</strong> most tweets are neutral (52.5% and 59.4%), and the Trump-search tweets come out <em>slightly more positive than negative</em> (26.1% vs 21.4%).</li></ul><p>The two tools give the same label on only <strong>58.3%</strong> of tweets. On 365 tweets, VADER says negative while TextBlob says positive. On 168, the opposite.</p><p>So which one is right? Reading the disagreements is the most useful hour you can spend on a sentiment project. A few patterns:</p><ul><li><strong>Sarcasm:</strong> “America! Are we great yet?” followed by a complaint about people being taken off the streets. VADER scores it +0.66, TextBlob +0.8. Both are wrong. The word “great” carries the score.</li><li><strong>Negation and comparison:</strong> “No president would be able to destroy as you have.” TextBlob sees no strong negative word and calls it positive.</li><li><strong>Short insults:</strong> “Vulgar man, small hands. Not an inch of heart.” VADER sees “heart” and leans positive.</li><li><strong>Nicknames:</strong> of 12 tweets using “sleepy Joe”, VADER called only 3 negative. “No cheating sleepy Joe!” scores +0.32. A lexicon doesn’t know a nickname is an insult.</li></ul><p>Neither tool is broken. They do exactly what they were designed for, which is scoring words. Political replies are full of sarcasm, nicknames and attacks phrased as questions, so the words alone often point the wrong way.</p><p>That’s the real lesson from my 2020 project: <strong>the choice of sentiment tool alone changed the headline</strong>. With TextBlob, the replies to both candidates look broadly similar. With VADER, the Trump-search replies look much more hostile. If you report a sentiment number without saying how it was produced, and without checking a sample by hand, you’re reporting the tool as much as the data.</p><h3>Part 2: What are people talking about?</h3><p>Sentiment says how people feel. Topics say about what. I used TF-IDF plus NMF (non-negative matrix factorisation), which is fast, needs no GPU and gives readable topics:</p><pre>from sklearn.feature_extraction.text import TfidfVectorizer<br>from sklearn.decomposition import NMF</pre><pre>txt = d.clean.str.replace(r&quot;@\w+&quot;, &quot;&quot;, regex=True).str.replace(&quot;#&quot;, &quot;&quot;)<br>vec = TfidfVectorizer(stop_words=&quot;english&quot;, min_df=5, max_df=0.4, ngram_range=(1, 2))<br>X = vec.fit_transform(txt)<br>nmf = NMF(n_components=8, random_state=0, init=&quot;nndsvd&quot;).fit(X)<br>d[&quot;topic&quot;] = nmf.transform(X).argmax(axis=1)</pre><p>Eight topics came out. The clearest ones:</p><ul><li><strong>Mail-in and absentee voting</strong>: 386 tweets, 56% negative. The most negative real policy topic.</li><li><strong>Delaying the election</strong>: the largest group by far, at 2,090 tweets. In practice it’s also a catch-all for general reactions.</li><li><strong>“Joe Biden / sleepy Joe / Obama”</strong>: 457 tweets, only 23% negative by VADER, which is suspicious given the nickname problem above.</li><li><strong>Insults and accusations of racism</strong>: 143 tweets, 73% negative.</li></ul><p>A simple keyword check gives the same picture: 770 tweets mention voting, mail or ballots, compared with 181 mentioning COVID and 28 about drug prices. In that week, the conversation was about the election process itself.</p><p>Hashtags were close to useless. Only 7.5% of tweets had one, and the most common tag appeared 19 times in 5,328 tweets. People replying to a politician write sentences, not hashtags. A hashtag-only analysis would have missed nearly all of this conversation.</p><h3>Part 3: Where LLMs actually help</h3><p>Everything above failed in predictable ways: sarcasm, nicknames, and not knowing <em>who</em> the negativity is aimed at. “@JoeBiden Your party cannot be trusted” and “@JoeBiden they are attacking you again, stay strong” both mention Biden. One is against him, one is for him, and a tweet-level sentiment score can’t tell you which.</p><p>This is where a large language model earns its cost. Instead of asking “is this tweet positive or negative?”, you ask a more useful question: <strong>what is the stance toward a specific target, and why?</strong></p><pre>PROMPT = &quot;&quot;&quot;You are labelling replies to political posts for a research study.<br>Reply to: {target}<br>Tweet: {text}</pre><pre>Return JSON only:<br>{{&quot;stance_to_target&quot;: &quot;support&quot; | &quot;oppose&quot; | &quot;neutral&quot; | &quot;unclear&quot;,<br>  &quot;sarcasm&quot;: true | false,<br>  &quot;topic&quot;: one of [&quot;voting process&quot;, &quot;covid&quot;, &quot;economy&quot;, &quot;law and order&quot;,<br>                    &quot;personal attack&quot;, &quot;other&quot;],<br>  &quot;reason&quot;: &quot;&lt;15 words max&gt;&quot;}}&quot;&quot;&quot;</pre><pre>def label_with_llm(client, model, target, text):<br>    msg = client.messages.create(<br>        model=model, max_tokens=150,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: PROMPT.format(target=target, text=text)}])<br>    return json.loads(msg.content[0].text)</pre><p>To be clear about what I did and didn’t do: <strong>I haven’t put LLM accuracy numbers in this post</strong>, because I didn’t run a properly labelled benchmark on this dataset. Any number I gave you would be a guess. What I can tell you is how I’d use it on a real project, and how I’d check it:</p><ol><li><strong>Hand-label a sample first.</strong> 200 to 300 tweets, labelled by two people, with disagreements discussed. This is your ground truth, and it also shows you how hard the task really is. If two humans agree only 80% of the time, no model will reliably do better.</li><li><strong>Run the lexicon and the LLM on the same sample.</strong> Compare both against the human labels. Report the agreement, not just “the LLM looks better”.</li><li><strong>Send the cheap cases to the cheap model.</strong> Use VADER for clearly positive or negative tweets with no sarcasm cues, and send the rest (questions, nicknames, mixed signals, disagreement between tools) to the LLM. On this dataset, the 41.7% of tweets where VADER and TextBlob disagree are a natural shortlist.</li><li><strong>Keep the “reason” field.</strong> It costs a few tokens and makes errors easy to spot when you review the output.</li><li><strong>Use the LLM to name things, not just label them.</strong> Giving it the top terms and five example tweets per NMF topic produces far better topic names than I would write by hand at 11 pm.</li></ol><p>LLMs also bring their own problems on political data. They can lean toward a default view, refuse borderline content, or label the same tweet differently on different runs. Fixing the temperature, keeping the prompt neutral and re-checking against the human sample handles most of this. On data about real elections, the human-labelled sample isn’t optional.</p><h3>Part 4: Network analytics: who talks to whom</h3><p>Text analysis treats each tweet on its own. Network analysis looks at connections. The simplest network in social data is the <strong>mention graph</strong>: each account is a node, and an edge from A to B means A mentioned B.</p><pre>import networkx as nx<br>from networkx.algorithms import community</pre><pre>G = nx.DiGraph()<br>for user, mentions in zip(d.user, d.mentions):<br>    for m in set(mentions):<br>        if G.has_edge(user, m): G[user][m][&quot;weight&quot;] += 1<br>        else: G.add_edge(user, m, weight=1)</pre><pre>top_in  = sorted(G.in_degree(), key=lambda x: -x[1])[:10]          # most mentioned<br>between = nx.betweenness_centrality(G.to_undirected(), k=600, seed=1)  # bridges</pre><p>The graph has 4,524 accounts and 5,446 edges. One connected component holds 4,500 of them, which is typical for replies: almost everyone is linked through the candidates’ accounts.</p><p>The most-mentioned accounts are the obvious ones: @JoeBiden (2,281 accounts mentioned him), @realDonaldTrump (1,895), @BarackObama (491), @FoxNews (309) and @DHSgov (141). On <strong>betweenness centrality</strong>, which measures how often an account sits on the shortest path between others, the two candidates dominate (0.73 and 0.65), and @FoxNews comes third at 0.08. It’s the main account connecting otherwise separate parts of the conversation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lRS0otJpjwbGf7f0lrA1zg.png" /></figure><p>The more interesting graph is the <strong>co-mention graph</strong>: two accounts are linked when they’re mentioned in the same tweet. I ran community detection (greedy modularity) on it:</p><pre>C = nx.Graph()<br>for mentions in d.mentions:<br>    for a, b in itertools.combinations(sorted(set(mentions)), 2):<br>        w = C[a][b][&quot;weight&quot;] + 1 if C.has_edge(a, b) else 1<br>        C.add_edge(a, b, weight=w)</pre><pre>groups = community.greedy_modularity_communities(C, weight=&quot;weight&quot;)<br>print(len(groups), community.modularity(C, groups, weight=&quot;weight&quot;))</pre><p>179 accounts, 366 links, 11 communities, and a modularity of 0.50, which means a clearly divided structure. The four largest groups are easy to recognise:</p><ul><li><strong>Biden cluster (74 accounts):</strong> @JoeBiden, @BarackObama, @ewarren, @PeteButtigieg, @BernieSanders.</li><li><strong>Trump cluster (53):</strong> @realDonaldTrump, @FoxNews, @DHSgov, @WSJ, @WhiteHouse.</li><li><strong>News organisations (19):</strong> @CNN, @nytimes, @AP, @latimes, @MiamiHerald and others, tagged together when people pulled the press into replies.</li><li><strong>Commentators and politicians (11):</strong> @SpeakerPelosi, @maddow, @washingtonpost, @KamalaHarris and others.</li></ul><p>And the number that best sums up the conversation: <strong>only 30 of 5,328 tweets mention both candidates</strong>. The two sides weren’t talking to each other. They were replying in parallel threads.</p><h3>The trap in mention networks</h3><p>Here’s what I missed in 2020, and what I now check first on any reply dataset.</p><p>The strongest link in the co-mention graph is @JoeBiden and @BarackObama, in 530 tweets. The next ones are @realDonaldTrump, @FoxNews and @DHSgov, each pair about 150 times. It’s tempting to read that as “people associate Biden with Obama” or “people connect Trump with Fox News”.</p><p>That’s mostly wrong. On Twitter, a reply automatically includes the accounts tagged in the post being replied to. When a candidate’s post tags another account, every reply to it inherits that tag. So these links mostly show <strong>which posts people replied to</strong>, not who they chose to bring into the conversation.</p><p>Two ways to handle it:</p><ul><li><strong>Remove inherited mentions.</strong> If you have the API’s reply metadata, strip out the leading @mentions that come from the reply chain and keep only the ones typed in the body of the tweet.</li><li><strong>Model the thread, not just the mention.</strong> Build a reply graph (who replied to which post) and a separate “chosen mention” graph. They answer different questions.</li></ul><p>The same trap exists in brand monitoring. If a company’s post tags a partner, every complaint underneath “mentions” the partner too. Report that as “negative mentions of the partner” and you’ve created a problem that isn’t there.</p><h3>Putting text and network together</h3><p>The real value comes from combining the two. For each major account, I took the tweets that mentioned it and looked at their VADER sentiment:</p><ul><li>Tweets mentioning @realDonaldTrump: 42.4% negative, 24.9% positive.</li><li>Tweets mentioning @JoeBiden: 25.6% negative, 29.6% positive.</li><li>Tweets mentioning @BarackObama: 19.4% negative, 36.6% positive.</li><li>Tweets mentioning @DHSgov: 38.7% negative, only 14.0% positive.</li><li>Tweets mentioning @FoxNews: 36.7% negative, 21.0% positive.</li></ul><p>Remember the earlier warning: these are lexicon scores on tweets, not stance toward the account. A negative tweet that mentions @JoeBiden may be attacking his opponent. That’s exactly why the stance-to-target LLM step matters. Combined with the network, it lets you report something much more precise, along the lines of: “inside the cluster around this post, 60% of replies oppose the target, mostly on the voting-process topic, and the accounts bridging to the other cluster are mostly news organisations.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7OjmX1cZ1bBZV9b5LkC4Sw.png" /></figure><h3>What I’d tell anyone starting a social media analytics project</h3><ul><li><strong>Check dates and collection method first.</strong> My own README was off by three months.</li><li><strong>Never trust a single sentiment tool.</strong> Run two, read the disagreements, and hand-label a sample. On this data, the tool choice alone flipped the headline.</li><li><strong>Measure stance toward a target, not just tone.</strong> This is where LLMs are worth paying for.</li><li><strong>Don’t lean on hashtags.</strong> Here they covered 7.5% of tweets.</li><li><strong>Separate inherited mentions from chosen ones</strong> before you draw conclusions from a network.</li><li><strong>Small samples aren’t public opinion.</strong> 5,328 replies collected over a few days tell you about the people who replied, not about voters. Report it that way.</li></ul><p>The tools here are all free and run on a laptop: pandas, VADER, TextBlob, scikit-learn and networkx. The LLM layer is the only paid part, and with the routing above you only send the hard cases to it.</p><p>I’ve used the same approach for media monitoring and political social media analytics in client work, with Power BI as the reporting layer: one page for volume and sentiment, one for topics, one for the network and key accounts. The method is the same whether the subject is an election, a product launch or a brand crisis.</p><blockquote><strong><em>Disclaimer:</em></strong><em> This analysis is for academic and educational purposes only. It does not represent the views of any candidate, party or organisation, makes no claim about public opinion or election outcomes, and should not be used for political targeting or campaigning. Tweets were collected via the public Twitter API in 2020, and individual users are not identified.</em></blockquote><p>Related posts:</p><ul><li><a href="https://gauravkrvish.medium.com/build-a-rag-powered-data-analytics-assistant-using-python-a9f111fa11bc">Build a RAG-Powered Data Analytics Assistant Using Python</a></li><li><a href="https://gauravkrvish.medium.com/building-an-ai-powered-executive-reporting-assistant-f773fd42da12">Building an AI-Powered Executive Reporting Assistant</a></li></ul><p>I build analytics, dashboards and practical machine learning for teams. You can see my work at <a href="https://iamgaurav.netlify.app/">iamgaurav.netlify.app</a>.</p><h4>Sources</h4><ul><li>Hutto and Gilbert, <a href="https://github.com/cjhutto/vaderSentiment">VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text</a>, ICWSM (2014)</li><li><a href="https://networkx.org/documentation/stable/reference/algorithms/community.html">NetworkX: community detection</a> documentation</li><li>Al Jazeera, <a href="https://www.aljazeera.com/amp/news/2020/7/30/trump-floats-election-delay-claims-voter-fraud-risks-amid-virus">Trump floats election delay, claims voter fraud risks amid virus</a> (30 July 2020)</li><li>6abc, <a href="https://6abc.com/trump-tweet-suburban-lifestyle-dream-suburb/6340805">Trump tweet on the “suburban lifestyle dream”</a> (July 2020)</li><li>WYPR/NPR, <a href="https://www.wypr.org/2020-07-28/trump-gives-medical-stockpile-a-kodak-moment-with-new-loan-to-make-drugs">Kodak loan to make drug ingredients</a> (28 July 2020)</li><li>Dataset and original notebook: <a href="https://github.com/GAURAV19999/USA-Election-Sentiment-Analysis">GAURAV19999/USA-Election-Sentiment-Analysis</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b32a1cbc9760" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recommendation Systems Explained: How Netflix and Amazon Do It, and How to Build One in Python]]></title>
            <link>https://gauravkrvish.medium.com/recommendation-systems-explained-how-netflix-and-amazon-do-it-and-how-to-build-one-in-python-4d5ae6033f66?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d5ae6033f66</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[netflix]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Thu, 05 Nov 2026 03:31:01 GMT</pubDate>
            <atom:updated>2026-11-05T03:31:01.552Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c9WJfwtBYg3U2yQWOWZdVg.png" /></figure><p>You’ve used at least one recommendation system today. The row of shows on Netflix, the “you might also like” strip on a shopping site, the next video that starts on YouTube. They’re so normal now that it’s easy to forget how much depends on them.</p><p>Netflix put numbers on it. In a 2015 paper, two of its executives wrote that recommendations influence the choice for about <strong>80% of hours streamed</strong> on the service, and estimated that personalisation and recommendations together save the company <strong>more than $1 billion a year</strong>. They also cited research that a typical member loses interest after perhaps 60 to 90 seconds of choosing. If the right title isn’t on screen by then, they leave.</p><p>This post covers how recommendation systems actually work, using three well-documented real examples, and then builds four of them in Python on a real public dataset of 921,398 movie ratings. I’ll show the evaluation numbers, the actual recommendations for real users, and a few results that surprised me.</p><h3>Three real examples</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uxAYsrVfsgtmnCNAP3vocw.png" /></figure><p><strong>Amazon: “people who liked this also liked”.</strong> In 2003, Amazon engineers Greg Linden, Brent Smith and Jeremy York published how the site’s recommendations worked. Instead of finding customers similar to you, which is slow with millions of customers, they built a table of <em>similar items</em>: products that the same customers tend to buy together. That table is computed offline. When you visit, the site only looks up items similar to the ones you’ve bought, so the online step stays fast however large the catalogue gets. This is called <strong>item-to-item collaborative filtering</strong>, and it’s still one of the best places to start.</p><p><strong>Netflix: personalisation everywhere.</strong> The 2015 paper describes not one algorithm but many: a personalised video ranker, rows like “Because you watched”, top picks, trending, and personalised search. The whole home page is assembled for each member. The lesson is that a “recommendation system” in practice is a collection of models, each answering a different question.</p><p><strong>YouTube: two stages.</strong> A 2016 Google paper on YouTube’s deep learning recommender describes a classic two-step design. A <em>candidate generation</em> model narrows millions of videos to a few hundred, then a separate <em>ranking</em> model orders those few hundred for you. Most large systems today follow this split, because it lets a cheap model do the wide search and an expensive model do the careful ordering.</p><h3>The four main approaches</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d78uGr4fo79hTZzVEnKb6Q.png" /></figure><ul><li><strong>Popularity:</strong> recommend what most people like. No personalisation, but surprisingly hard to beat, and the standard fallback for new users.</li><li><strong>Content-based:</strong> recommend items with similar attributes to what you liked, such as the same genre, brand or skills. It works for brand-new items but tends to recommend more of the same.</li><li><strong>Collaborative filtering:</strong> recommend based on behaviour. Either “users like you liked this” (user-based) or Amazon’s “people who liked this also liked that” (item-based).</li><li><strong>Matrix factorisation:</strong> learn a small set of hidden “taste” dimensions for every user and item, and score how well they match. This family became famous during the Netflix Prize competition.</li></ul><p>Real systems are usually <strong>hybrids</strong>: collaborative filtering where there’s enough data, content and popularity where there isn’t.</p><h3>The dataset: real ratings from real people</h3><p>I used <strong>MovieTweetings</strong>, a public research dataset built by Simon Dooms and colleagues at Ghent University. It collects movie ratings that people posted publicly on Twitter in a standard format, like “I rated The Martian 8/10” with an IMDb link. The version I downloaded has 921,398 ratings on a 1–10 scale.</p><p>Three preparation steps, all of which matter more than the choice of algorithm:</p><ul><li><strong>Remove thin history.</strong> I kept users and movies with at least 20 ratings each. That left 593,397 ratings from 8,576 users on 4,779 movies.</li><li><strong>Define “liked”.</strong> Most real systems work with signals like clicks or purchases, not star ratings. I treated a rating of 8 or more as “liked”, which turns this into the same kind of problem a shop or streaming service has.</li><li><strong>Split by time, not at random.</strong> For each user, their last 20% of ratings became the test set. The model only sees the past and has to predict what they liked next. A random split lets the model peek at the future and makes every result look better than it really is.</li></ul><pre>r = pd.read_csv(&quot;data/ratings.dat&quot;, sep=&quot;::&quot;, engine=&quot;python&quot;,<br>                names=[&quot;user&quot;, &quot;item&quot;, &quot;rating&quot;, &quot;ts&quot;])<br>for _ in range(5):                                   # repeat until stable<br>    r = r[r.groupby(&quot;item&quot;).user.transform(&quot;size&quot;) &gt;= 20]<br>    r = r[r.groupby(&quot;user&quot;).item.transform(&quot;size&quot;) &gt;= 20]<br>r[&quot;liked&quot;] = r.rating &gt;= 8</pre><pre>r = r.sort_values(&quot;ts&quot;)<br>r[&quot;rank&quot;] = r.groupby(&quot;user&quot;).cumcount(ascending=False)<br>r[&quot;n&quot;] = r.groupby(&quot;user&quot;).item.transform(&quot;size&quot;)<br>test_mask = r[&quot;rank&quot;] &lt; (r[&quot;n&quot;] * 0.2).clip(lower=1)  # each user&#39;s latest 20%<br>train, test = r[~test_mask], r[test_mask &amp; r.liked]</pre><p>The training data becomes a sparse user × movie matrix with a 1 wherever a user liked a movie. It’s 8,576 × 4,779, and almost entirely empty, which is normal.</p><h3>Building the four recommenders</h3><p><strong>Popularity</strong> counts how many people liked each movie:</p><pre>pop = np.asarray(X.sum(axis=0)).ravel()      # likes per movie<br>scores = pop.copy()                           # same list for everyone</pre><p><strong>Item-item collaborative filtering</strong> is Amazon’s idea. Two movies are similar if the same people liked them, measured with cosine similarity. A user’s score for a movie is the sum of its similarity to everything they liked:</p><pre>Xn = normalize(X.T.tocsr(), axis=1)           # one row per movie<br>S = (Xn @ Xn.T).tocsr()                       # movie-to-movie similarity<br>S.setdiag(0)<br>scores = X[user] @ S                          # sum of similarities to what they liked</pre><p>The similarity table S is computed once, offline, exactly as the Amazon paper describes. Scoring a user is just a lookup and a sum.</p><p><strong>Matrix factorisation</strong> compresses the matrix into a small number of hidden dimensions with truncated SVD. Each user and each movie gets a vector, and the score is how well they line up:</p><pre>svd = TruncatedSVD(n_components=32, random_state=0)<br>U = svd.fit_transform(X)                      # users  x 32<br>V = svd.components_                           # 32 x movies<br>scores = U[user] @ V</pre><p><strong>Content-based</strong> uses only genres: it builds a TF-IDF genre profile from the movies a user liked and scores every movie by how close its genres are.</p><p>For every model, movies the user has already rated are removed, and the top 10 are recommended.</p><h3>How to measure a recommender</h3><p>For each of the 8,321 users who liked at least one movie in both periods, I compared the top 10 recommendations with the movies they actually went on to like in the test period:</p><ul><li><strong>Hit rate@10:</strong> the share of users for whom at least one of the 10 recommendations was a movie they later liked. This is the easiest metric to explain to a business audience.</li><li><strong>Precision@10:</strong> how many of the 10 were right, on average.</li><li><strong>Recall@10:</strong> what share of the movies they later liked were caught in the top 10.</li><li><strong>Coverage:</strong> what share of the whole catalogue ever gets recommended to anyone. Low coverage means the system keeps pushing the same few titles.</li></ul><h3>The results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TlpoQly8YZ50h1eL4hceZw.png" /></figure><ul><li><strong>Item-item collaborative filtering: 27.2% hit rate</strong>, precision 3.9%, coverage 29.1%.</li><li><strong>Matrix factorisation (SVD, 32 dimensions): 25.3% hit rate</strong>, coverage 8.7%.</li><li><strong>Popularity: 19.3% hit rate</strong>, coverage 1.3%.</li><li><strong>Content-based (genres only): 3.9% hit rate</strong>, coverage 69.3%.</li></ul><p>A precision of 3.9% sounds tiny, but remember the task: pick 10 movies out of 4,779 that a specific person will later rate 8 or more, using only their past. Picking 10 movies at random would give a precision of about 0.15% and a hit rate of about 1.5%. Small absolute numbers are normal for this kind of offline test. What matters is the comparison between models.</p><p>Four things stood out.</p><p><strong>1. Popularity is a serious baseline.</strong> Recommending the same blockbusters to everyone already hits for almost one user in five. Any model you build has to beat this, and some published “improvements” don’t. But popularity covers only 1.3% of the catalogue, so the rest of your inventory never gets seen.</p><p><strong>2. The simple Amazon-style method won.</strong> Item-item collaborative filtering beat popularity by about 40% on hit rate, and covered over 20 times more of the catalogue. It’s also the easiest to explain, which matters more than people think.</p><p><strong>3. The fancier model needed tuning and still lost.</strong> SVD scored 25.0% with 16 dimensions, 25.3% with 32, 24.7% with 64 and only 21.9% with 128. More dimensions isn’t automatically better. Matrix factorisation can win with more tuning, regularisation or a method designed for implicit feedback like ALS, but out of the box it didn’t beat a similarity table.</p><p><strong>4. Content-based was weak on accuracy, but not useless.</strong> Genres alone are too broad: thousands of movies are “Drama”. But it recommended 69% of the catalogue and it’s the only one of the four that can recommend a brand-new movie nobody has rated yet. That’s its job in a hybrid system.</p><h3>What the recommendations actually look like</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oyNSMW5JIqQStVynuC--rQ.png" /></figure><p>Numbers are only half the story. Here are real item-item recommendations for one user from the dataset, with the “because you liked” explanation taken from the movies that contributed most to each score.</p><p>This user had liked 33 movies in the training period, including <em>X-Men</em>, <em>Minority Report</em>, <em>Batman Begins</em>, <em>Gravity</em> and <em>The Wolf of Wall Street</em>. The top five recommendations were:</p><ol><li><em>The Hunger Games: Catching Fire</em>, because they liked <em>Gravity</em> and <em>Frozen</em></li><li><em>X-Men: Days of Future Past</em>, because they liked <em>The Wolf of Wall Street</em> and <em>Gravity</em> → <strong>they did go on to like it</strong></li><li><em>Captain Phillips</em>, because they liked <em>Gravity</em> and <em>Prisoners</em></li><li><em>Captain America: The Winter Soldier</em>, because they liked <em>The Wolf of Wall Street</em> and <em>Gravity</em> → <strong>they did go on to like it</strong></li><li><em>The Hobbit: The Desolation of Smaug</em>, because they liked <em>Gravity</em> and <em>The Wolf of Wall Street</em></li></ol><p>Two hits in the top five. But look at the explanations. <em>The Wolf of Wall Street</em> doesn’t have much to do with <em>Captain America</em>. What they share is <strong>timing</strong>: people in this dataset tweet ratings for what they’re watching now, so movies released around the same time get liked by the same people. Collaborative filtering doesn’t know about genres or plots. It learns whatever pattern is in the behaviour, including “what’s in cinemas this season”.</p><p>That’s a strength (it finds patterns no one tagged) and a warning (explanations can look strange to users, and the model will reflect any quirk in how your data was collected).</p><h3>The cold-start problem</h3><p>Collaborative filtering needs history. Two situations break it:</p><ul><li><strong>A new user</strong> with no likes yet. The standard fix is popularity, ideally popularity within a segment the user chose (a country, a category, a “pick three movies you like” screen).</li><li><strong>A new item</strong> that nobody has interacted with. Only content-based methods can recommend it, using its attributes. This is why the weakest model in my test still earns a place in a real system.</li></ul><h3>Beyond movies: where this is used in business</h3><p>The same four approaches show up everywhere once you look:</p><ul><li><strong>Retail and e-commerce:</strong> “frequently bought together”, next-best product, basket completion.</li><li><strong>B2B and banking:</strong> next-best offer for a relationship manager, based on what similar clients use.</li><li><strong>HR and learning:</strong> recommending courses based on what people in similar roles completed, or internal jobs based on skills.</li><li><strong>Content and media:</strong> articles, podcasts, reports in a BI portal (“people who viewed this dashboard also used…”).</li></ul><p>In most of these, item-item collaborative filtering on purchase or click data is a sensible first version. It can be built in a day with the code above, and it gives you a baseline that any fancier model has to beat.</p><h3>Common mistakes</h3><ul><li><strong>Random train/test splits.</strong> Use time-based splits. Otherwise you’re testing whether the model can predict the past.</li><li><strong>No popularity baseline.</strong> Without it you can’t tell whether a complex model is actually adding anything.</li><li><strong>Optimising only accuracy.</strong> Track coverage and diversity too, or the system slowly collapses onto the same few items.</li><li><strong>Feedback loops.</strong> Users click what you show them, which becomes training data, which makes you show it more. Keep some exploration in the mix.</li><li><strong>Trusting offline numbers too much.</strong> Offline tests rank models. Only an online A/B test tells you whether people actually watch, buy or stay more.</li></ul><h3>Wrapping up</h3><p>A recommendation system doesn’t have to start with deep learning. The ideas behind the Netflix, Amazon and YouTube systems work at any scale: learn from behaviour, use content and popularity where behaviour is missing, and always measure against a simple baseline with an honest, time-based test. On real data, a similarity table built in a few lines of Python beat both popularity and an untuned matrix factorisation model, and it can explain every recommendation it makes.</p><p>Related posts:</p><ul><li><a href="https://gauravkrvish.medium.com/i-built-an-ai-data-analyst-from-natural-language-to-sql-to-business-insights-831539717fa3">I Built an AI Data Analyst: From Natural Language to SQL to Business Insights</a></li><li><a href="https://gauravkrvish.medium.com/build-a-rag-powered-data-analytics-assistant-using-python-a9f111fa11bc">Build a RAG-Powered Data Analytics Assistant Using Python</a></li><li><a href="https://gauravkrvish.medium.com/building-an-ai-powered-executive-reporting-assistant-f773fd42da12">Building an AI-Powered Executive Reporting Assistant</a></li></ul><p>I build analytics, dashboards and practical machine learning for teams. You can see my work at <a href="https://iamgaurav.netlify.app/">iamgaurav.netlify.app</a>.</p><h4>Sources</h4><ul><li>Gomez-Uribe and Hunt, <a href="https://dl.acm.org/doi/10.1145/2843948">The Netflix Recommender System: Algorithms, Business Value, and Innovation</a>, ACM Transactions on Management Information Systems (2015)</li><li>Linden, Smith and York, <a href="https://www.cs.umd.edu/~samir/498/Amazon-Recommendations.pdf">Amazon.com Recommendations: Item-to-Item Collaborative Filtering</a>, IEEE Internet Computing (2003)</li><li>Covington, Adams and Sargin, <a href="https://research.google/pubs/pub45530/">Deep Neural Networks for YouTube Recommendations</a>, ACM RecSys (2016)</li><li>Dooms, De Pessemier and Martens, <a href="https://github.com/sidooms/MovieTweetings">MovieTweetings: a Movie Rating Dataset Collected From Twitter</a>, CrowdRec at RecSys (2013)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d5ae6033f66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI vs Generative AI: What’s the Difference, and When to Use Which]]></title>
            <link>https://gauravkrvish.medium.com/agentic-ai-vs-generative-ai-whats-the-difference-and-when-to-use-which-219a0d959a4e?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/219a0d959a4e</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 03:31:01 GMT</pubDate>
            <atom:updated>2026-11-04T03:31:01.775Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XYnX5Rnp0kefGfcbDjEw-w.png" /></figure><p>A year ago, every AI pitch was about generative AI. Now every pitch says “agents”.</p><p>Clients ask me the same question in different words: “Is this agentic thing different, or is it just a new label?” The honest answer is both. Agentic AI is a genuinely different way of using AI. It’s also being slapped onto plenty of products that are nothing of the sort.</p><p>In this post I’ll explain the difference in plain language, show the same business task done both ways, and give a simple guide for when each one makes sense, from the point of view of someone who works with data every day.</p><h3>Generative AI: you ask, it creates</h3><p>Generative AI is what most people mean by “AI” today: ChatGPT, Claude, Gemini, Copilot. You give it a prompt, it gives you something back. Text, code, an image, a summary, a SQL query.</p><p>The key thing is that <strong>you drive every step</strong>. It answers, you decide what to do with the answer, then you ask again. It’s like a very fast, very well-read assistant who only acts when spoken to and never leaves their desk.</p><p>That makes generative AI great at:</p><ul><li>Drafting: emails, reports, documentation, code</li><li>Explaining: an error message, a contract clause, a DAX formula</li><li>Summarising: survey comments, long documents, meeting notes</li><li>Transforming: rewriting, translating, reformatting</li></ul><h3>Agentic AI: you set a goal, it works toward it</h3><p>An AI agent starts with a <strong>goal</strong>, not a prompt. It then works out the steps, uses tools to carry them out (run a database query, call an API, read a file, send an email), looks at what came back, and decides what to do next. It keeps going until the goal is met or it hits a limit.</p><p>Under the hood, an agent still uses a generative model to think. The difference is what surrounds it: tools, memory of what it has already done, and permission to act without a human typing the next instruction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fC-tDzsCvNJeX2dEo-H-pQ.png" /></figure><p>The row I’d pay most attention to is <strong>main risk</strong>. A generative AI mistake is a wrong answer you can read and ignore. An agent mistake can be a wrong action that has already happened: an email sent, a record updated, a refund issued.</p><h3>How an agent actually works</h3><p>Almost every agent follows the same loop:</p><ol><li><strong>Plan:</strong> break the goal into steps.</li><li><strong>Act:</strong> call a tool to do one step.</li><li><strong>Observe:</strong> read the result.</li><li><strong>Reflect:</strong> is this right? Is the goal done? Should it try something else?</li></ol><p>Then it goes round again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6TR6lN8_9WStSSsN0ZXAtg.png" /></figure><p>A good agent also has <strong>guardrails</strong> around that loop: which tools it may use, what data it can see, how many steps or how much money it can spend, and which actions need a human to approve them first.</p><p>When I <a href="https://gauravkrvish.medium.com/i-built-an-ai-data-analyst-from-natural-language-to-sql-to-business-insights-831539717fa3">built a small AI data analyst</a>, it sat somewhere in between. It wasn’t a full agent, but it had a small loop: if its SQL was blocked by the validator, the error went back to the model to fix and try again. That one feedback step made it noticeably more reliable. Full agents take that idea much further.</p><h3>The same task, two ways</h3><p>Here’s an illustrative example. A sales director asks: <em>“Find out why North region sales fell last month and tell the regional manager.”</em></p><p><strong>With generative AI,</strong> you do the work and AI speeds up each step. You ask it for SQL, run the query yourself, paste the results back, ask for a summary, check it, and send the email.</p><p><strong>With an agent,</strong> you hand over the goal. It plans a few queries (sales by product, by customer, by week), runs them on a read-only database, spots that one large customer stopped ordering, drafts an email with a chart, and stops to ask for your approval before sending.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D7hlafvBrra34hfIIxfP9w.png" /></figure><p>Same outcome. The difference is who does the steps. With generative AI, you’re the one moving things along. With an agent, you’re the one checking and approving.</p><h3>Where the hype is ahead of reality</h3><p>Agents are exciting, but it’s worth being realistic about where things stand.</p><p>In June 2025, Gartner predicted that <strong>over 40% of agentic AI projects will be cancelled by the end of 2027</strong>, because of rising costs, unclear business value or weak risk controls. It also called out “agent washing”: vendors relabelling chatbots, assistants and RPA tools as agentic. Gartner estimated that only about <strong>130 of the thousands of vendors</strong> claiming agentic AI were the real thing (<a href="https://w.media/over-40-percent-of-agentic-ai-projects-could-face-the-axe-by-end-of-2027-gartner/">W.Media, reporting Gartner</a>).</p><p>At the same time, Gartner expects that by 2028, at least <strong>15% of day-to-day work decisions</strong> will be made autonomously by agentic AI (up from none in 2024), and <strong>33% of enterprise software applications</strong> will include agentic AI (up from less than 1% in 2024).</p><p>So both things are true. Agents are coming into real software fast, and a lot of today’s agent projects won’t survive. The ones that do will be the ones solving a specific, well-defined problem.</p><h3>When to use which</h3><p><strong>Use generative AI when:</strong></p><ul><li>A human needs to review the output anyway (reports, emails, code)</li><li>The task is one step: draft, explain, summarise, rewrite</li><li>Mistakes are cheap and easy to spot</li><li>You’re still figuring out what the right process even is</li></ul><p><strong>Consider an agent when:</strong></p><ul><li>The task has many steps that follow a predictable pattern</li><li>It repeats often enough that automating the whole flow pays off</li><li>The tools and data it needs are clean and well-defined</li><li>You can clearly define “done” and “wrong”</li><li>Risky actions can be kept behind a human approval step</li></ul><p><strong>Use neither when</strong> a simple rule or a scheduled report would do the job. A lot of “agent” projects are a cron job with extra steps.</p><h3>If you do build an agent: five guardrails</h3><ol><li><strong>Least access.</strong> Give it only the tools and data it needs. Read-only by default.</li><li><strong>Human approval for anything irreversible.</strong> Sending, paying, deleting, publishing.</li><li><strong>Limits.</strong> Maximum steps, maximum cost, maximum time per task.</li><li><strong>Logs.</strong> Record every step it took and why, so you can review and debug.</li><li><strong>Test cases.</strong> A set of tasks with known correct outcomes, re-run whenever anything changes.</li></ol><h3>What this means for data teams</h3><p>For analysts, I think the practical change is this: generative AI made each of our tasks faster. Agents will start doing whole sequences of tasks, the pull-check-summarise-share routine that fills a lot of reporting work.</p><p>That doesn’t remove the analyst. Someone still has to define the metrics, build a clean data model the agent can trust, decide which actions need approval, and check that the agent’s conclusions make sense. If anything, agents make that groundwork more important. An agent running on messy data and vague definitions just makes mistakes faster and at scale.</p><h3>The short version</h3><p>Generative AI answers. Agentic AI acts. Most businesses should get very good at the first before they hand over the second, and when they do, they should start small, with clear limits and a human in the loop.</p><p>If you’re trying to work out where generative AI or agents fit into your reporting and analytics, I’m happy to talk it through. You can find me at <a href="https://iamgaurav.netlify.app/">iamgaurav.netlify.app</a>.</p><h4>Sources</h4><ul><li>W.Media, <a href="https://w.media/over-40-percent-of-agentic-ai-projects-could-face-the-axe-by-end-of-2027-gartner/">Over 40 percent of Agentic AI projects could face the axe by end of 2027: Gartner</a> (June 27, 2025), reporting Gartner’s press release of June 25, 2025</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=219a0d959a4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AI-Powered Executive Reporting Assistant]]></title>
            <link>https://gauravkrvish.medium.com/building-an-ai-powered-executive-reporting-assistant-f773fd42da12?source=rss-2eb74bda54fe------2</link>
            <guid isPermaLink="false">https://medium.com/p/f773fd42da12</guid>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Gaurav Kumar Vishvakarma]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 13:31:00 GMT</pubDate>
            <atom:updated>2026-11-03T13:31:00.085Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sGVV4CdFkJAXYaCMq4rMcw.png" /></figure><p>Every month, somewhere in most companies, an analyst spends two days building a deck that a CEO reads in four minutes. Pull the numbers, check them against finance, compare to last year, find the two things that actually matter, write three sentences about them, and format it all so it fits on one page.</p><p>It’s an obvious job for AI. It’s also a dangerous one. An executive summary is the one document where a wrong number gets repeated in a board meeting, and where a confident sentence like “driven by strong online demand” gets taken at face value.</p><p>In this post I build an executive reporting assistant in Python that produces a one-page monthly brief: KPI tiles, a headline, three bullets, a decision line and two charts. The design rule is simple and it shaped everything else: <strong>code calculates the numbers, AI only writes the words, and a checker compares the two before anyone sees it.</strong></p><p>The data is a synthetic sales dataset (2,600 orders across 2024 and 2025), so the numbers are real outputs, not made-up examples.</p><h3>Why not just ask the AI to write the report?</h3><p>You can paste a spreadsheet into a chat model and ask for an executive summary. The result usually reads well. The problems are underneath:</p><ul><li><strong>It does arithmetic in its head.</strong> Growth rates and shares get calculated by the model, and occasionally they’re slightly wrong.</li><li><strong>It fills gaps with plausible stories.</strong> If the data doesn’t say why revenue rose, the model often supplies a reason anyway.</li><li><strong>It reaches for superlatives.</strong> “Record month”, “strongest performance ever”. Sometimes true, often not checked.</li><li><strong>It’s not repeatable.</strong> Run it twice and the numbers might be rounded differently, or a different point leads.</li></ul><p>For a report that goes to leadership, every number needs to be traceable to a query. So the architecture splits the work.</p><h3>The architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QoCCInB9daj9aU-Tzo2ORw.png" /></figure><ol><li><strong>Facts:</strong> Python and SQL compute every KPI, comparison and driver, and save them as a JSON file.</li><li><strong>Narrative:</strong> an LLM writes the headline, bullets and decision line from that JSON, and only from that JSON.</li><li><strong>Verification:</strong> code checks that every number in the text exists in the facts, and flags unsupported claims.</li><li><strong>Render:</strong> the brief is built as HTML and PDF.</li><li><strong>Review:</strong> a human approves it before it’s sent.</li></ol><h3>Step 1: Compute the facts in code</h3><p>The facts layer answers the questions an executive always asks. How did we do this month? Compared to last month, last year and plan? What about the year so far? What drove the change?</p><pre>def kpis(d):<br>    rev, prof, orders = d.Sales.sum(), d.Profit.sum(), d.OrderID.nunique()<br>    return {&quot;revenue&quot;: round(rev, 0), &quot;profit&quot;: round(prof, 0),<br>            &quot;margin_pct&quot;: round(100 * prof / rev, 1),<br>            &quot;orders&quot;: int(orders), &quot;aov&quot;: round(rev / orders, 0)}</pre><pre>def build_facts(month):<br>    df = load()<br>    m = pd.Period(month, &quot;M&quot;)<br>    cur, prev_month, last_year = df[df.month == m], df[df.month == m - 1], df[df.month == m - 12]<br>    k, kl = kpis(cur), kpis(last_year)<br>    return {<br>        &quot;period&quot;: str(m),<br>        &quot;kpis&quot;: k,<br>        &quot;vs_last_year_pct&quot;: {x: pct(k[x], kl[x]) for x in (&quot;revenue&quot;, &quot;profit&quot;, &quot;orders&quot;, &quot;aov&quot;)},<br>        &quot;ytd&quot;: ytd_comparison(df, m),<br>        &quot;drivers_vs_ly&quot;: {&quot;category&quot;: drivers(cur, last_year, &quot;Category&quot;),<br>                          &quot;region&quot;:   drivers(cur, last_year, &quot;Region&quot;),<br>                          &quot;channel&quot;:  drivers(cur, last_year, &quot;Channel&quot;)},<br>        ...<br>    }</pre><p>The drivers function is the most useful part. It takes the change in revenue versus last year and splits it by category, region and channel, so the report can say which segment actually moved the number:</p><pre>def drivers(cur, prev, dim):<br>    a = cur.groupby(dim).Sales.sum()<br>    b = prev.groupby(dim).Sales.sum()<br>    delta = a.subtract(b, fill_value=0).round(0)<br>    return [{&quot;name&quot;: k, &quot;change&quot;: float(v)}<br>            for k, v in delta.sort_values(key=abs, ascending=False).items()]</pre><p>For December 2025, the facts file says: revenue $82,548, up 51.6% on December 2024. Profit up 68.3%. Margin 28.5%, up 2.8 points. Electronics added $26,135. The Retail channel added $28,468 while Online fell $377. And year to date, revenue is $740,518, which is <strong>2.8% below 2024</strong>.</p><p>That last number is the one a dashboard often buries and an executive most needs to see.</p><h3>Step 2: Let the AI write, from facts only</h3><p>The prompt is short and strict. It gives the model a fixed structure and forbids it from calculating or explaining anything that isn’t in the JSON:</p><pre>PROMPT = &quot;&quot;&quot;You write the monthly executive summary for the leadership team.<br>Use ONLY the facts in the JSON below. Do not calculate new numbers, do not round<br>differently, and do not explain causes that are not in the facts.</pre><pre>Write exactly:<br>1. Headline: one sentence, the single most important point.<br>2. Three bullets: what happened, what drove it, what to watch.<br>3. One &quot;decision needed&quot; line, or &quot;No decision needed this month.&quot;<br>Keep it under 120 words. Plain English, no jargon, no exclamation marks.</pre><pre>FACTS:<br>{facts}&quot;&quot;&quot;</pre><p>The “decision needed” line is my favourite part of the format. It forces the summary to answer the question every executive is silently asking: <em>do I need to do anything?</em> Most months the honest answer is no, and saying so explicitly saves a meeting.</p><p>I also built a rule-based writer that produces the same structure with fixed sentence templates. It’s the fallback when the model is unavailable, and it’s what produced the brief in this article. In production I’d use both: the LLM for better wording, and the template version as a sanity check.</p><h3>Step 3: Verify every number before anyone reads it</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vhuv-EfLIfZ95So64uOUrg.png" /></figure><p>This is the step that makes the whole thing trustworthy. The checker pulls every number out of the summary and confirms it appears in the facts file, either as-is or in a K/M format. It also flags superlatives that the facts don’t support:</p><pre>def verify(text, facts, derived=()):<br>    ok = allowed_numbers(facts) | {round(d, 1) for d in derived}<br>    problems = []<br>    for raw in re.findall(r&quot;\d[\d,]*\.?\d*&quot;, text):<br>        v = float(raw.replace(&quot;,&quot;, &quot;&quot;))<br>        if v in {2024, 2025, 2026} or v &lt;= 3:<br>            continue<br>        if round(v, 1) not in ok and round(v, 2) not in ok:<br>            problems.append(raw)<br>    claims = re.findall(r&quot;\b(record|best ever|all-time|highest ever|unprecedented)\b&quot;, text, re.I)<br>    return {&quot;unverified_numbers&quot;: problems, &quot;unsupported_claims&quot;: claims,<br>            &quot;passed&quot;: not problems and not claims}</pre><p>To test it, I wrote a draft with the kind of mistakes models make:</p><p><em>“December was a record month: revenue hit $82.5K, up 61% year on year, driven by strong online demand. Margin reached 31.2%.”</em></p><p>The checker flagged <strong>61</strong> (the real figure is 51.6), <strong>31.2</strong> (the real margin is 28.5) and <strong>“record”</strong>. The draft would have been blocked.</p><p>But look at what it <em>didn’t</em> catch: “driven by strong online demand”. Online revenue actually <strong>fell</strong> $377. Growth came entirely from Retail. The sentence contains no wrong number, so a number checker can’t see the problem. That’s the honest limit of automated verification, and it’s why the last step is a person, not a script.</p><p>You can push the checker further, for example by flagging any segment name that appears next to “driven by” or “thanks to” unless it’s a top driver in the facts. But some judgement will always need a human.</p><h3>Step 4: Build the one-page brief</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B0edkEwA_SZa3ZVsYioZ1g.png" /></figure><p>The brief is plain HTML, rendered to PDF with a headless browser. It has five KPI tiles, the headline and bullets, the decision line, a 13-month revenue trend and a driver chart. A badge at the top says “Numbers verified” or “Needs review”, depending on the checker’s result.</p><p>Here’s the December 2025 summary it produced:</p><p><strong>December revenue was $82.5K, up 51.6% on last year, but the full year is still 2.8% behind 2024.</strong></p><ul><li>Profit rose 68.3% and margin improved 2.8 points to 28.5%.</li><li>Electronics added $26.1K, about 93% of the increase; Retail added $28.5K while Online slipped $0.4K.</li><li>Watch Furniture: margin down 3.2 points on last year.</li></ul><p><em>No decision needed this month. Review the 2026 outlook with January results before changing plans.</em></p><p>The verifier passed it: zero unverified numbers, zero unsupported claims. The only number not in the facts file, 93%, is calculated in code and passed to the checker as a derived value.</p><h3>What a good executive summary actually says</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-rge5L6P_jCII3V6GKd_gQ.png" /></figure><p>The interesting thing about December isn’t the 51.6%. It’s the contrast. December was the best month in the last 13, but the year as a whole finished 2.8% down. A summary that only celebrated December would leave the leadership team with the wrong impression of the year.</p><p>That’s why the facts layer always includes the year-to-date comparison, and why the headline is built to lead with the single most important point, not the best news. A strong month that doesn’t fix the year is the headline a good analyst would write.</p><p>The second point is concentration. Electronics delivered about 93% of the year-on-year increase, and the Retail channel delivered all of it. When one category and one channel explain nearly all the growth, that’s both the story and the risk.</p><h3>Step 5: Human review and delivery</h3><p>The assistant never sends anything by itself. The workflow I’d use:</p><ul><li>The brief is generated on the second working day, after finance closes the month.</li><li>The analyst who owns the report reviews it, edits the wording if needed, and approves.</li><li>Only then is it emailed or posted. The approval, the facts file and the final text are saved together.</li><li>If the checker says “Needs review”, the brief can’t be approved until the flagged numbers are fixed.</li></ul><p>That review takes 10 minutes instead of two days of building, and the analyst’s time goes into judgement rather than formatting.</p><h3>Lessons from building it</h3><ul><li><strong>Numbers come from code, words from AI.</strong> Never let the model do the arithmetic for a report that matters.</li><li><strong>Always include the year-to-date view.</strong> One strong month is the most common way an executive summary misleads.</li><li><strong>Show what drove the change</strong>, not just the change. Driver analysis is cheap to compute and it’s what executives actually ask about.</li><li><strong>Verify, then review.</strong> The checker catches wrong numbers. A person catches wrong reasoning.</li><li><strong>Keep the format fixed.</strong> Same tiles, same structure, every month. Executives learn where to look.</li><li><strong>Label assumptions.</strong> The plan in my facts file is illustrative (last year plus 5%), and the brief says so in the footer.</li></ul><h3>Wrapping up</h3><p>The value of an AI reporting assistant isn’t that it writes nicely. It’s that it turns a two-day monthly chore into a 10-minute review, without lowering the bar on accuracy. That only works if the numbers are computed, the words are constrained, and someone still reads it before it goes out.</p><p>Related posts:</p><ul><li><a href="https://gauravkrvish.medium.com/i-built-an-ai-data-analyst-from-natural-language-to-sql-to-business-insights-831539717fa3">I Built an AI Data Analyst: From Natural Language to SQL to Business Insights</a></li><li><a href="https://gauravkrvish.medium.com/build-a-rag-powered-data-analytics-assistant-using-python-a9f111fa11bc">Build a RAG-Powered Data Analytics Assistant Using Python</a></li><li><a href="https://gauravkrvish.medium.com/ai-in-hr-reporting-what-it-should-automate-and-what-it-shouldnt-e10ab446de5a">AI in HR Reporting: What It Should Automate, and What It Shouldn’t</a></li></ul><p>I build executive dashboards, reporting automation and practical AI tools for teams. You can see my work at <a href="https://iamgaurav.netlify.app/">iamgaurav.netlify.app</a>.</p><p><em>All data in this article is synthetic and generated with Python.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f773fd42da12" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>