What the Research Actually Says About Semantic Layers and Text-to-SQL Accuracy

Seven studies, what each one measured, what none of them tested, and the numbers that keep getting misquoted.

The gains are large. The samples are small.

Most claims about semantic layers and text-to-SQL accuracy trace back to a handful of studies. I went back to the ones that get cited most. They agree on the direction: giving a model written business context raises accuracy a lot, and more than switching models does. But they are small, mostly single-dataset, often run by a vendor, and they don’t test the same thing. In one study “semantic layer” means a compiled metrics engine. In another it means a 4 KB markdown file pasted into the prompt.

Here is what each study measured, what it didn’t, and how the numbers get stretched on their way into slide decks.

The seven studies, briefly

GSK’s pharmacovigilance study (JAMIA Open, 2025)

GSK researchers asked GPT-4 to turn 60 real questions from user logs into Oracle SQL against a drug-safety database. With the full schema (a 290-page PDF) fed through a retrieval pipeline, 8.3% of answers passed. Adding a business context document raised that to 78.3%, per the paper in JAMIA Open.

The detail most summaries skip is the third phase. Narrowing the schema to the 33 essential tables, without the context document, produced a 10% pass rate. Shrinking the schema barely helped. Explaining the business did.

The authors list the limits themselves: one model, one proprietary database, 60 questions. You will sometimes see this cited as “the Bayer study.” It’s GSK’s.

Cube’s paired benchmark (2026)

This one often appears twice in reading lists, as a Cube blog post and as arXiv 2604.25149. It is one study, written by Cube’s VP of Product and a co-author. It ran 100 questions on Contoso retail data in ClickHouse across Claude Opus 4.7, Claude Sonnet 4.6 and GPT-5.4, single-shot, with and without a 4 KB hand-written markdown document describing measures, conventions and disambiguation rules.

Accuracy rose 17 to 23 points on every model, to between 67.7% and 68.7%. The more interesting result: within each condition, the three models were statistically indistinguishable. The document explained the variance. The model choice didn’t.

What it did not test is a semantic layer engine. Nothing was compiled to SQL. It’s strong evidence for semantic content, which is a broader and different claim.

dbt Labs’ 2026 benchmark

dbt Labs reran its 2023 test on the ACME Insurance dataset: 11 questions, 20 runs each, with Claude Sonnet 4.6 and GPT-5.3 Codex.

• In scope: the dbt Semantic Layer scored 100% on both models. Text-to-SQL scored 62.5% and 51.2%.
• Questions needing too many entity hops: the semantic layer scored 0%, because it refused. Text-to-SQL scored 70.0% and 100.0% on those.
• All 11 questions: 72.7% for the semantic layer, 64.5% for text-to-SQL on both models.
• After an LLM added three dbt models: the semantic layer reached 98.2% and 100%. Text-to-SQL reached 90.0% and 84.1%.

The authors flag the main caveat themselves: to make text-to-SQL work they loaded the entire schema as context, “which isn’t practical for larger datasets.” It’s also 11 questions on one dataset, run by the vendor of the winning approach.

Spider 2.0 (ICLR 2025)

Spider 2.0 is 632 real enterprise text-to-SQL workflow tasks. In the November 2024 paper, an agent framework built on o1-preview solved 21.3% of them, against 91.2% on the older Spider 1.0 and 73.0% on BIRD.

That 21.3% is still quoted as if it described today’s models. It doesn’t. The top entry on the Spider 2.0-Snow leaderboard now scores 96.70. And Spider 2.0 was never a semantic layer study. It measured how hard raw enterprise schemas are, and the leaderboard since then shows how much agents that explore metadata and documentation can close that gap.

Sequeda, Allemang and Jacob (2023)

GPT-4, zero-shot, on an enterprise insurance schema: roughly 16% accuracy when asking the SQL database directly, roughly 54% when asking over a knowledge graph built with an ontology and mappings (arXiv 2311.07509). The insurance benchmark it introduced is the one dbt reused in 2026. It’s the origin of the “context triples accuracy” argument, and it predates several generations of models.

Snowflake’s semantic model study (2025)

On four BIRD subsets with Claude 3.5 Sonnet, agent-improved semantic models (relationships, descriptions, custom instructions) lifted average accuracy from 57% to 78%. Two caveats from the post itself: the comparison bundles the semantic model with Cortex Analyst’s agent system, so the gain can’t be credited to the model alone, and answers were graded by an LLM judge.

The DABStep ablation (2026)

This is the most useful study for separating causes. Qing Ye’s ablation held tools, retrieval instructions and the table allow-list fixed, then emptied the prose out of a YAML data contract. Semantic content lifted hard-task accuracy from 13.9% to 55.1% on the weakest of four models and from 37.0% to 77.4% on the strongest. Scaffolding alone added 0 to 5 points on two smaller models and 14 to 15 on two frontier models.

Get Ka Ling Wu’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The contract also beat pasting the same knowledge into the prompt: generated SQL carried the fee rules up to 98% of the time, against 4% for the prompt version. Ungoverned runs issued 166 data-modifying statements. Governed runs issued none.

Accuracy without and with business context, as reported by each study. Datasets differ, so compare within a row, not across rows. Sources: JAMIA Open, Cube, Snowflake, arXiv 2609.22259, dbt Labs.

Where the studies agree

Context moves accuracy more than model choice. Cube found frontier models indistinguishable once they had the same document. GSK got from 8.3% to 78.3% without changing the model at all.

Meaning is the active ingredient, not schema size or plumbing. GSK’s trimmed schema barely moved. Ye’s scaffolding-only arm added far less than content. Even plain column descriptions help only a little on their own: an Alibaba study on BIRD found average accuracy across three open models went from 36.25% with no descriptions to 37.18% with generated ones and 38.59% with hand-written ones. Descriptions name things. Business context explains them.

Compiled layers change how you fail. The dbt semantic layer refused what it couldn’t model instead of guessing. In dbt’s words, “Text-to-SQL will cheerfully give you a wrong number.”

What none of them shows

A fair fight between compiled and written context. dbt compared its semantic layer against text-to-SQL with a raw schema. Cube compared a markdown document against a raw schema. Neither compared a compiled layer against an equally rich written document on the same questions. The closest is Ye’s study, where pre-computed views served as a ceiling, and the gap between the written contract and that ceiling shrank from 39 points to 5 as models got stronger.

Scale. The headline studies use 11, 60 and 100 questions, each on a single dataset.

Coverage in the wild. A compiled layer’s 100% applies to the questions it can answer. Nobody has published what share of a real company’s question queue falls in scope.

Upkeep. GSK notes that context documents need ongoing maintenance as schemas change. None of the studies measure what happens when the context goes stale.

Seven studies, their setups and the caveat that travels with each headline.

How the numbers get misquoted

• “LLMs only solve 21% of enterprise SQL.” That’s a November 2024 baseline. The Spider 2.0-Snow leaderboard top score is now 96.70.
• “A semantic layer is 100% accurate.” On the in-scope questions of an 11-question set, after it refused the rest.
• “Two independent studies, from Cube and arXiv, agree.” They’re the same study.
• “The Bayer study.” The pharmacovigilance paper is GSK’s.
• “Descriptions over 20 words hurt accuracy.” In the Alibaba paper, the 20-word cap is a design choice motivated by that claim, not a measured result.
• “Semantics improve agent accuracy by 80%.” That’s a Gartner prediction for 2027 (“up to 80%”), not a measurement.

Questions to ask the next benchmark

New studies will keep arriving. These are the questions I now ask before quoting one:

1. How many questions, and were they taken from real logs or written for the test?
2. Who ran it, and do they sell the approach that won?
3. What exactly was “the semantic layer”: a compiled engine, a YAML file or a markdown document?
4. What did the baseline get: the full schema, retrieved snippets or nothing?
5. How were refusals scored?
6. Single-shot prompt, or an agent that could explore the data first?
7. Who graded the answers: execution match, a human or an LLM judge?

My reading of the evidence: write down business meaning first, because that’s what every study rewards, and compile the metrics that must be exact, because that’s what changes the failure mode. I go deeper on what goes into a semantic layer and on the context layer around it in two longer guides.

Follow me on Medium and LinkedIn for more on analytics agents, context engineering, and the practical infrastructure behind production-grade AI.