Why AI Data Agents Still Struggle With Enterprise Data

Photo by Zheng Yang on Unsplash

96.7% on the leaderboard, 34.8% on the job

AI data agents now score close to perfect on the best-known text-to-SQL benchmark. On a new benchmark built to look like a real enterprise warehouse, the best of 14 models fully solves only about a third of the tasks.

The top score on Spider 2.0-Snow has climbed from 23.8% at release to 96.7%. Read quickly, that looks like enterprise data work is nearly solved. Argo-Bench, published on 1 October 2026, puts agents inside a 235-table ERP warehouse with 7.5 billion rows and asks them to do what analysts actually do: investigate, decide and act. The strongest model scores 95 or more on just 34.8% of tasks (Tomitsuka et al., 2026).

The gap is not about writing SQL. The SQL usually runs. The failures are quieter: the agent answers a slightly different question than the one asked, and does it carefully and confidently.

This article walks through two recent papers that explain where data agents fail, what the failures have in common, and Python code you can use to catch the same failures in your own agent before a stakeholder does.

What makes enterprise data hard for agents

A data agent is an LLM that explores a database, writes and runs queries or Python, and returns an answer or takes an action. Most benchmarks test the middle step, turning a question into SQL. Real enterprise work is harder at both ends: finding the right data, and deciding what the question actually means.

The Argo-Bench authors point to three gaps between benchmarks and real warehouses:

• Public, simple data. Spider 2.0-Snow’s 547 questions spread across 152 databases, mostly public datasets whose schemas appear in tutorials. In an enterprise warehouse, a single business event can touch ten or more tables that must agree with each other.
• Not end to end. Real tasks need forecasting, fraud models and optimisation, then a decision. A correct number is not the same as a good decision.
• Unreliable answer keys. A 2026 audit found errors in 62.8% of Spider 2.0-Snow’s released gold queries and 52.8% of BIRD Mini-Dev’s, and fixing BIRD’s errors moved leaderboard rankings by up to nine places (Jin et al., 2026).

Anyone who has worked on a large data platform will recognise the hardest part: business definitions. Who counts as an active member? When does a cancelled contract stop counting? Which of three tables is the source of truth for revenue? These answers usually live in people’s heads, not in the schema.

Paper 1: Argo-Bench — careful analysis of the wrong thing

Argo-Bench’s main finding is that failing agents often use sound methods but read the wrong record, optimise the wrong objective or measure the wrong quantity (Tomitsuka et al., 2026). The paper comes from TextQL and was posted on 1 October 2026.

The setup

The authors simulated a New York City food delivery platform for 2024 at real scale, calibrated to public city reports and company filings, and exported it to an Oracle E-Business Suite warehouse.

The key design choice is that the simulator’s true state is hidden from the warehouse. The agent must reconstruct the facts from the tables, while the grader reads them directly. That makes tasks easier to check than to solve, just like real analytics.

The results

The best model, Claude Opus 5.5, fully solved 34.8% of tasks and averaged 59.5 points out of 100. Nine of the 14 models averaged below 35.

Three ways careful agents get it wrong

• Wrong record. A dashboard task needed to know who counted as a member when an offer was sent. Members whose card is declined keep benefits for a seven-day grace period, but their contract stays on the books for weeks. Twenty-four failing runs took membership from the contract dates. Seven of them got every other figure right to within a few dollars. The passing runs replayed the billing history and checked it against orders where a member benefit was actually applied.
• Wrong objective. A task asked where to cut $9.6 million from courier bonuses. One model ran a sophisticated analysis to cut where bonuses bought the fewest courier-hours, and lost money, scoring 0. Another worked out that bonuses exist to avoid surge pay, cut where they saved the least surge, and captured $3.09 million of an attainable $3.12 million. When the prompt explained what bonuses are for, the average score rose from 18.5 to 61.7.
• Wrong quantity. In one task, 46 of 47 completed runs missed every monthly count, because the warehouse keeps hourly check-ins while the app sends them every half hour. Across dashboard tasks, 66.9% of data sources that passed their structural checks scored zero on their actual values.

One more finding is worth knowing: across 4,553 forecast series, intervals labelled as 80% contained the real value only 44.8% of the time. Data agents are overconfident about uncertainty too.

Get Aarthy Ramachandran’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The common thread is the most useful lesson in the paper: passing runs checked their definitions against a second source.

Paper 2: Data Intelligence Agents — the SQL runs, the answer is still wrong

The second paper reaches the same conclusion from the opposite direction. It builds a high-scoring data agent, then studies what is left when it fails: almost every failed query runs to completion and returns a wrong answer (Vyas et al., 2026). The system, from C3 AI, was posted on 17 June 2026 and is deployed with enterprise customers.

How the agent works

DIA (Data Intelligence Agents) uses one autonomous coding agent in three roles: a Data Interpreter that profiles raw sources, a Schema Creator that builds and validates a database, and a Query Generator that answers questions. All three share a memory of past work. The Query Generator follows four steps:

1. Declare the shape first. Before writing SQL, write down the expected result: which columns, one row per what, what order, which filters.
2. Probe, don’t guess. Run small queries to confirm join keys exist, sample real values to learn their format, and check row counts.
3. Generate and run the query.
4. Check the result against the declared shape. If it does not match, diagnose, fix and rerun before answering.

With one model (Claude Sonnet 4.5) and no fine-tuning, this matched or beat the best published result on all seven SQL benchmarks it was tested on. The biggest gain was on conversational tasks, BIRD-Interact, where it scored 55.7% against a previous best of 22.7%.

What still goes wrong

• Reasoning failures dominate. The largest class of errors is a query that answers a subtly different question: a wrong join, filter or formula on an under-specified request. Queries that fail to run at all are under 3% of failures.
• Hypothesis lock-in. On BIRD-Interact, 81% of failures used up the full 15-turn budget re-submitting nearly identical SQL instead of stopping to ask a clarifying question. The authors conclude the decisive skill is knowing when to stop guessing and ask.
• Self-checks share the agent’s blind spot. The shape check is written by the agent itself. If it misreads the question, the query and the check inherit the same mistake, and a wrong answer passes.
• Composite metrics are hardest. The agent was weakest on questions that name a business metric without spelling out its formula.

A learned rule worth stealing

The shared memory records rules only when the data confirms them. One example: counting schools through a one-to-many join, COUNT(*) returned 9,977 for a single school, while COUNT(DISTINCT school_id) returned 1. The rule it learned: when counting entities through a repeating join, count distinct primary keys.

How the two papers fit together

Both papers say the bottleneck has moved from syntax to semantics. Argo-Bench shows careful agents reading the wrong record or optimising the wrong goal. DIA shows that even a state-of-the-art agent mostly fails by answering a slightly different question, and that checking its own work only helps if the check is grounded in something other than its own reading of the question.

Hands-on: four guardrails for your data agent

The failures in both papers can be caught with simple checks wrapped around whatever agent you use. The code below uses only the Python standard library and SQLite, so you can run it as-is and then point it at Snowflake, Databricks or Postgres through your own connection. Every sample was run before publishing.

# guards.py
import difflib
import sqlite3
from dataclasses import dataclass, field


# 1. Declare the expected shape before running the query

@dataclass
class ShapeContract:
    columns: list[str]   # columns the answer must have, in order
    grain: list[str]     # one row per ... (these columns must be unique)
    min_rows: int = 1
    max_rows: int | None = None
    not_null: list[str] = field(default_factory=list)


def check_shape(conn: sqlite3.Connection, sql: str, contract: ShapeContract) -> list[str]:
    """Run the query and return a list of problems (empty list = passes)."""
    cur = conn.execute(sql)
    cols = [d[0] for d in cur.description]
    rows = cur.fetchall()
    problems = []
    if cols != contract.columns:
        return [f"columns {cols} != expected {contract.columns}"]
    if len(rows) < contract.min_rows:
        problems.append(f"only {len(rows)} rows, expected at least {contract.min_rows}")
    if contract.max_rows is not None and len(rows) > contract.max_rows:
        problems.append(f"{len(rows)} rows, expected at most {contract.max_rows}")
    idx = [cols.index(c) for c in contract.grain]
    keys = [tuple(r[i] for i in idx) for r in rows]
    if len(keys) != len(set(keys)):
        problems.append(f"grain {contract.grain} is not unique")
    for c in contract.not_null:
        nulls = sum(r[cols.index(c)] is None for r in rows)
        if nulls:
            problems.append(f"{nulls} NULLs in {c}")
    return problems


# 2. Catch join fan-out: COUNT(*) vs COUNT(DISTINCT key)

def fanout_check(conn: sqlite3.Connection, from_clause: str, entity_key: str) -> dict:
    """If a join repeats entity rows, counting rows over-counts entities."""
    rows, entities = conn.execute(
        f"SELECT COUNT(*), COUNT(DISTINCT {entity_key}) {from_clause}"
    ).fetchone()
    return {"rows": rows, "distinct_entities": entities,
            "fanout": round(rows / entities, 2) if entities else None,
            "warning": rows != entities}


# 3. Check a definition against a second source

def cross_check(conn, name: str, sql_a: str, sql_b: str, tolerance: float = 0.01) -> dict:
    """Compute the same number two independent ways."""
    a = conn.execute(sql_a).fetchone()[0]
    b = conn.execute(sql_b).fetchone()[0]
    gap = abs(a - b) / max(abs(a), abs(b), 1)
    return {"metric": name, "source_a": a, "source_b": b,
            "relative_gap": round(gap, 3), "agree": gap <= tolerance}


# 4. Stop hypothesis lock-in: ask instead of resubmitting

class LockInGuard:
    """Flags when the agent keeps submitting near-identical SQL after failures."""

    def __init__(self, similarity: float = 0.9, max_repeats: int = 2):
        self.similarity = similarity
        self.max_repeats = max_repeats
        self.history: list[str] = []

    def submit(self, sql: str) -> str:
        normalised = " ".join(sql.lower().split())
        repeats = sum(
            difflib.SequenceMatcher(None, normalised, past).ratio() >= self.similarity
            for past in self.history
        )
        self.history.append(normalised)
        if repeats >= self.max_repeats:
            return "STOP: ask the user a clarifying question before retrying"
        return "ok"

Trying it on two classic bugs

The demo builds a tiny database with two traps from the papers. The first is a one-to-many join that inflates counts, like DIA’s 9,977-vs-1 example. The second is a membership table where one customer’s contract is still on the books after their card was declined, like Argo-Bench’s wrong-record failure.

# demo.py (abridged)
buggy = """SELECT s.county, COUNT(*) AS schools
           FROM schools s JOIN test_scores t ON s.school_id = t.school_id
           GROUP BY s.county"""
fixed = buggy.replace("COUNT(*)", "COUNT(DISTINCT s.school_id)")

contract = ShapeContract(columns=["county", "schools"], grain=["county"],
                         max_rows=10, not_null=["schools"])
print("shape check (buggy):", check_shape(conn, buggy, contract))
print("fan-out:", fanout_check(conn, "FROM schools s JOIN test_scores t "
                                     "ON s.school_id = t.school_id", "s.school_id"))
print("reconcile buggy:", cross_check(conn, "total schools",
      f"SELECT SUM(schools) FROM ({buggy})", "SELECT COUNT(*) FROM schools"))
print("cross-check:", cross_check(conn, "June members",
    "SELECT COUNT(*) FROM membership_contracts "
    "WHERE start_date <= '2026-06-30' AND end_date >= '2026-06-01'",
    "SELECT COUNT(DISTINCT customer_id) FROM orders "
    "WHERE member_benefit = 1 AND order_date LIKE '2026-06-%'"))

guard = LockInGuard()
for attempt in [buggy, buggy + " ", buggy.replace("county", "county ")]:
    print("lock-in guard:", guard.submit(attempt))

Output:

shape check (buggy): []
fan-out: {'rows': 6, 'distinct_entities': 3, 'fanout': 2.0, 'warning': True}
reconcile buggy: {'metric': 'total schools', 'source_a': 6, 'source_b': 3, 'relative_gap': 0.5, 'agree': False}
cross-check: {'metric': 'June members', 'source_a': 3, 'source_b': 2, 'relative_gap': 0.333, 'agree': False}
lock-in guard: ok
lock-in guard: ok
lock-in guard: STOP: ask the user a clarifying question before retrying

What the output teaches

• The shape check passed the buggy query. The columns, grain and row count were all fine. This is exactly DIA’s limitation: a shape check catches structural mistakes, not semantic ones.
• The fan-out and reconciliation checks caught it. Each school appeared twice on average after the join, and the per-county total (6) did not match the school table (3). After the fix, both sources agree on 3.
• The cross-check caught the wrong record. Contract dates say 3 members in June; orders where a benefit was actually applied say 2. Neither number is automatically right, but the gap tells the agent to stop and work out which definition the business means.
• The lock-in guard forced a question on the third near-identical attempt, instead of letting the agent burn its whole turn budget.

A checklist before a data agent touches enterprise data

• Write down business definitions. Who is a member, what counts as revenue, which table is the source of truth. Give the agent these, rather than hoping it infers them from column names.
• State the objective, not just the task. Argo-Bench scores rose from 18.5 to 61.7 when the prompt explained what a budget was for.
• Make the agent declare the result shape before it writes SQL, and check the result against it.
• Profile before querying. Sample real values and check join cardinality instead of trusting column names.
• Count distinct keys through joins, and flag any join where row count and entity count differ.
• Reconcile every important number with a second source, such as a control total, another table or another module.
• Cap retries and require a clarifying question after repeated near-identical attempts.
• Ask for intervals, then check their coverage. Agents’ 80% intervals covered the real value only 44.8% of the time.
• Keep a human review step for actions, such as bans, payouts and budget changes, until your own evaluation shows the agent is reliable on your data.
• Evaluate on your own warehouse. Public leaderboard scores don’t transfer, and public answer keys are often wrong themselves.

What I took away, in plain terms

• The bottleneck has moved from SQL to meaning. Queries almost always run. The failures are answers to a slightly different question.
• Careful is not the same as correct. Some of the worst failures in Argo-Bench used sophisticated methods on the wrong record or toward the wrong goal.
• Cross-checking is the habit that separated winners from losers. Passing runs checked their definitions against a second source.
• A self-check only helps if it looks somewhere new. An agent checking its own reading of the question will agree with itself.
• Asking is a skill. Most stuck agents kept guessing when one clarifying question would have saved them.
• Leaderboards are not your warehouse. A 96.7% benchmark score said little about how agents handle a real ERP schema.

None of this means data agents aren’t useful. It means the engineering around them matters as much as the model: definitions, reconciliation, clarifying questions and human review for actions. These are the same disciplines good data teams already practise. Agents need them written down.

If you’ve put a data agent on real enterprise data, what broke first: the SQL, or the definitions?

References

1. Tomitsuka, G., Raayatsanati, A., Xing, E., Gand, D. and Ma, J. J. (2026). Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows. arXiv:2610.02122. Full text: HTML. Code: GitHub. Data: Hugging Face.
2. Vyas, A., Dhanuka, A., Khoshfetrat Pakazad, S. and Ohlsson, H. (2026). Data Intelligence Agents: Interpreting, Modeling, and Querying Enterprise Data via Autonomous Coding Agents. arXiv:2606.19319. Full text: HTML.
3. Jin, T., Choi, Y., Zhu, Y. and Kang, D. (2026). Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards. arXiv:2601.08778. Cited via Argo-Bench.
4. Lei, F. et al. (2025). Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. ICLR 2025.