The Last Mile of Autonomous RCA: Database Access
How we gave AI agents free-form SQL access to live production databases without losing sleep.
AI agents have become genuinely good at investigating production issues. Point one at a bug report and it will read the code, pull the logs, check the dashboards, and walk the traces. Then it hits a wall, and it’s always the same wall: what does the data actually say?
A customer reports something broken. The fix usually starts with one row — the record’s status and its error code. The agent can’t get even that. So one of two things happens. Either the investigation stalls while a human runs SQL and pastes results back, turning every root-cause analysis into a ticket queue with extra steps. Or, worse, the agent doesn’t stall. It fills the gap with assumptions and produces a confident, well-written, plausible RCA that’s wrong.
Database access is the last mile of autonomous investigation. This post is about the challenges we hit giving agents that access to live production databases, and the solutions that have held up.
Why you can’t just hand over a connection string
Four problems, and each one is disqualifying on its own.
PII. Anything that enters a prompt leaves your boundary. It may be logged or retained, and you can’t reliably delete it back. Under GDPR and CCPA, “the AI read a customer’s email address while debugging” is not a sentence you want to say out loud.
Load. These are live production databases serving real customers. Our largest table has 30+ billion rows. One careless SELECT with a missing filter is an incident.
Writes. An autonomous agent with write access to prod is an incident waiting for a timestamp.
Prompts are not guardrails
The tempting fix is to write a very stern system prompt. You are a read-only assistant. Never query customer data. Keep queries cheap.
The failure modes are worth spelling out, because they shaped everything we built:
- A prompt is injectable. Any row, log line, or ticket the agent reads can carry instructions that override yours.
- A prompt is probabilistic. “Keep queries cheap” holds only as often as the model complies.
- A prompt fails irreversibly. Once data is in the context window, it’s out.
- A prompt is un-auditable. You can’t prove what it prevented, or what it didn’t.
The principle we landed on: enforcement must live in the database, not the prompt. Prompts still steer the agent — schema hints, error messages, suggestions. But steering sits on top of enforcement, never in place of it.
Three ways to do it
We found three workable architectures. They’re all good; the choice is your use case.
Point the agent at your data warehouse. If your databases already replicate into a warehouse with tag-based masking policies, this is the shortest path. Masking is enforced by the platform, governance lives in one place, and the agent never touches prod. The cost is freshness: replication lag makes it the wrong tool for “what is happening right now” questions.
Get Rahuljpgupta’s stories in your inbox
Join Medium for free to get updates from this writer.
Build a proxy service. A standing service between the agent and the databases: it validates queries, redacts results (which catches PII hiding in free-text columns, the one thing nothing else catches), and keeps a central audit log. Maximum control, but it’s the most engineering of the three — something you build, deploy, patch, and keep on-call for.
Give the agent a database role, and put policy in front of it. A dedicated read-only role whose column-level grants exclude PII, a validation layer in the agent’s tooling, and query transport over the same Kubernetes access engineers already have. Near-zero new infrastructure, live data, and enforcement by the database engine itself.
We needed live data and didn’t want to run another service, so we took the third path. Here’s how each of the four problems got solved.
PII: a permissions problem, not a prompt problem
The agent connects as a role that has SELECT on explicitly named safe columns and nothing else. Not "grant everything, then revoke the sensitive parts" — that's default-allow, and every new column ships exposed until someone remembers to revoke it. Explicit grants mean a new, unclassified column is born invisible. No prompt, injected or otherwise, can read a column the role can't see, because the query dies in the database engine with a permission error.
Classifying hundreds of columns by hand is the kind of chore that kills projects, so we didn’t do it by hand. An AI pass drafts the classification — allow, deny, or uncertain — for every column in the live schema. Every uncertain column gets an explicit human decision. The result is code-reviewed like any other change, and a human applies the grants. AI-first, human-verified. The AI never grants itself anything.
Load: restriction proportional to table size
A 5,000-row lookup table and a 30-billion-row events table are not the same risk, so they shouldn’t live under the same rules. Every table gets a tier from its live row count:
- Open (under 500k rows): scan freely, up to 1,000 rows back, 10s timeout.
- Anchored (500k–10M): the query must filter on an indexed column, 200 rows, 10s.
- Heavy (10M+): a direct indexed filter on the table itself, 50 rows, 5s.
Touch several tables in one query and the strictest tier governs — otherwise you could smuggle a huge scan into a query framed around a tiny table.
Static rules can’t see everything, though. A query can be perfectly shaped — indexed filter, small limit — and still be expensive, because the filtered value matches 40% of the table. The shape-checker can’t know that. The database’s query planner can. So before any statement executes, we run EXPLAIN on it and read the planner's own cost estimate. Over the tier's ceiling, and the query is rejected before it runs, with the measured cost in the refusal.
This isn’t theoretical. In live testing, a structurally valid query on a heavy table came back with an estimated cost of 140,124 against a ceiling of 100,000 and was denied pre-execution. The agent read the refusal, narrowed its filter, and succeeded on the next attempt. And if the planner ever underestimates, the statement timeout is still waiting.
That “read the refusal and adapted” part matters more than it looks. Every denial names the rule and suggests a fix. An agent that gets permission denied retries blindly; an agent told "this table requires an indexed filter — try filtering on one of these columns" rewrites correctly on the next attempt. Guardrails that teach are the difference between a safety system and a brick wall.
Where it landed
This has been running against our production fleet for a while now: free-form SQL over 500+ tables, the largest at 30+ billion rows, with the catalog of tables, tiers, and grants regenerated automatically from the database itself — nobody hand-maintains a list. Agents investigate real bugs end-to-end, and runaway queries aren’t discouraged; they’re structurally impossible.
If you’re considering the same road, the three things I’d carry over:
- Put enforcement in the database. Grants and gates are deterministic and injection-proof. Prompts are neither.
- Make restrictions proportional. Uniform strictness makes agents useless on small tables and still doesn’t protect the big ones well. Tier by measured size.
- Make guardrails teach. A denial that explains itself turns a blocked query into a corrected one.
The last mile is buildable. The data your agents need is sitting right there — you just have to let them in without letting everything out.
