Your AI governance stack has a hole in the middle
William Thompson (@iteratingself)
Financial services leads every industry in AI-agent adoption. Roughly 91% of institutions have agents doing real work, mostly in customer and account operations. Only about 21% of those same organizations call their governance for those agents mature (McKinsey, 2026). A bank examiner spends his time in that 70-point gap, and he doesn’t open with “is your model safe.”
He opens with a narrower, harder question. FINRA’s 2026 Annual Regulatory Oversight Report spells it out: can you reconstruct every action your agent took? Not summarize it. Reconstruct it: the decision, the escalation, the customer interaction, timestamped and attributable, on demand, for an account the agent touched eight months ago.
Most teams answer by pulling up a trace dashboard. A trace is a recording of what an agent improvised. It is not a control. That distinction is the subject of this piece.
A KYC agent, and the question nobody built for
Picture a bank that has put an AI agent in front of account opening. A new customer applies. The agent runs sanctions screening, pulls identity verification, checks address history, scores the account for risk, and either clears the account or escalates it to a human analyst. It calls five or six APIs to do this: an OFAC list provider, an ID-verification vendor, a credit bureau, an internal risk model, a case-management system.
Two customers apply on the same day with similar profiles. The agent doesn’t necessarily run their checks in the same order or call the same services, because the model decides the next step at inference time, call by call. Most of the time this doesn’t matter. Both accounts clear, and both customers are fine. Six months later an examiner picks one account and asks why the sanctions screen ran after the risk score instead of before it. Was that ordering policy, or an accident of that day’s inference pass? Nobody can answer with certainty, because the order was never a decision anyone made. It was an emergent property of a model run, decided fresh each time, never guaranteed to repeat, and reconstructable only from whatever the observability tool happened to log.
This is the tension underneath every 2026 governance framework. Agents improvise, and regulated industries are regulated precisely because improvisation is not allowed. Not discouraged. Not allowed. The rule exists because someone got hurt by a process that didn’t run the same way twice.
Why this got urgent this year
Four regulators and standards bodies moved the question this year, in the same direction, from four different jurisdictions.
The EU AI Act’s high-risk obligations were set to bite on August 2, 2026. In late June the Council approved the Digital Omnibus package, pushing the deadline for stand-alone high-risk systems to December 2, 2027. The sixteen months change the date, not the demand. Article 12 requires automatic event logging across the system’s lifecycle: six months’ retention minimum, twenty-four for biometric and law-enforcement use. Article 14 requires human oversight designed into the system, including two-person verification for certain classes of decision. Both articles reach the agent’s action layer directly: the API calls, the tool invocations, the MCP servers a KYC agent calls. The Cloud Security Alliance already calls most deployments an “enterprise readiness gap.” Logging that happens to occur through an observability integration is not logging as a system property.
US banking is almost the inverse. SR 26-2, the April 2026 update to model risk management from the Fed, OCC, and FDIC, explicitly excludes generative and agentic AI from its scope. The agencies signaled a future RFI and left it there. So the systems banks are deploying fastest, the agentic account-opening and servicing flows, sit in a supervisory vacuum. Trade press calls it the “kill-switch gap”: examiners show up with no framework written for what they’re looking at.
FINRA didn’t wait. Its 2026 report moved agentic AI from “emerging technology” to a named, active supervisory priority. The reconstruct-every-action language above is theirs.
Singapore’s IMDA published the first agentic-specific government framework in January 2026 and updated it in May. It’s voluntary, but it’s specific in a way most frameworks aren’t: bound risks upfront, keep humans meaningfully accountable, and enforce technical controls through the agent’s lifecycle, explicitly including whitelisted services. Not “monitor what the agent does.” Whitelist what it’s allowed to do in the first place.
ISO/IEC 42001, the AI management system standard, is in its first real certification wave. BSI, A-LIGN, Schellman, and KPMG are all running audits, and Fortune 500 procurement has started asking vendors for the certificate or a credible roadmap to one. Its Annex A.9 requires defined escalation paths for agent deployments. That is the same demand as EU Article 14, arriving from the standards side instead of the regulatory side.
Four bodies, four jurisdictions, one convergent demand: stop governing the model in the abstract and start bounding, overseeing, and reconstructing what the agent actually did.
What the bank has already bought
None of this caught the market flat-footed. If our KYC bank went shopping for “AI governance” this year, it would find a real, mature vendor ecosystem, and it would come home with something in each of four categories.
The paper layer handles policy and documentation. Credo AI, Holistic AI, IBM watsonx.governance, and OneTrust translate regulatory obligations into policy registries and audit-ready documentation. That is a real and necessary function, especially for a 42001 audit or an EU Act conformity file. But what it produces is a description of what should happen. Enforcement is delegated to whatever actually runs the agent.
Get Iteratingself’s stories in your inbox
Join Medium for free to get updates from this writer.
The watch layer handles observability. Arthur, Fiddler, and agent-tracing tools like LangSmith and Langfuse give you traces, evals, and drift monitoring across every call the agent made. This has real value. It is how the bank would even notice that the sanctions screen ran out of order. But it is a record of what did happen, assembled after it happened, with no power to have made it happen differently.
The guardrail layer sits inline. AI gateways, Microsoft’s Agent 365 and Entra Agent ID, ServiceNow’s AI Control Tower, and policy-gate products like ElixirData’s filter the agent’s calls as they go by: block a call to a disallowed endpoint, flag a suspicious parameter. This is progress over the watch layer, because it can actually stop something. But it filters an improvised stream of actions rather than removing the improvisation, and it works from a deny-list. It catches what someone thought to disallow, applied call by call. The audit story is per-call too, not per-workflow.
The durability layer handles execution. Temporal, AWS Step Functions, Restate, and agent-graph tools like LangGraph give you deterministic, replayable execution. The workflow won’t silently die if a pod restarts mid-run. But the workflow is developer code, not an artifact anyone outside engineering can review or sign off on. And the replay log is an implementation detail of the execution engine, not a tamper-evident record designed for an auditor to read.
Four layers, four categories of vendor, each solving a real problem well. The bank buys all four and still cannot answer the examiner’s question about why the sanctions screen ran second.
The gap is structural, not incompetence
This is not vendor incompetence. Each layer is honest about what it does: document, observe, filter, execute. Each does that job well. The gap exists because none of the four decides, ahead of time, what the agent is allowed to do and in what order, and then makes that the only thing that can happen.
There is a real difference between control flow as an emergent property of an inference pass and control flow as a reviewed, versioned, hash-verified artifact. All four layers are built to accommodate the first: observe it, filter it, log it, run it durably. None of them is built to replace it with the second.
I have watched this exact split before, in a different regulated context: governing where cloud workloads were allowed to run. Which region, which jurisdiction, which data-residency boundary. That problem went through the same two generations of tooling. First came a dashboard that told you a workload had drifted outside its approved boundary. Later, and separately, came a policy engine that made the drift impossible to deploy in the first place. Watching and preventing were never the same product, and the watching tool always shipped first, because you can bolt it onto an existing pipeline without changing how anyone deploys. Agent governance is running the same sequence, a few years behind.
The IAPP’s AI Governance Vendor Report 2026 makes this concrete without meaning to. It segments the market into four categories: policy and compliance, technical assessments, assurance and auditing, consulting and advisory. Every one of those categories is paper, measurement, or attestation. There is no runtime-enforcement segment. The taxonomy doesn’t have a box for “make the compliant behavior the only behavior available.”
What the missing layer has to actually provide
Line the regulatory demands up next to each other and they stop looking like separate requirements. They are one requirement, stated four ways.
IMDA’s whitelisted-services requirement and EU Article 14’s designed-in oversight want the same thing: a declared, closed catalog of permitted actions, not a menu the agent is trusted to choose from responsibly. For the KYC agent, that means the sanctions screen, the ID check, the risk score, and the escalation path exist as a fixed sequence someone reviewed and signed off before the first real customer ran through it. The same sequence every time. Not a fresh improvisation per applicant.
FINRA’s reconstruct-every-action language and EU Article 12’s automatic logging want the same thing too: an execution record that is a first-class, tamper-evident resource, not a side effect of whichever observability tool is plugged in. The academic literature is converging here as well. “Auditable Agents” (arXiv 2604.05485) names five dimensions an execution record needs (action recoverability, lifecycle coverage, policy checkability, responsibility attribution, evidence integrity) and measures the cost of tamper-evident, pre-execution mediation at about 8.3 milliseconds. That number matters because performance is the usual objection to this kind of enforcement, and 8.3ms against an API call that is already waiting on a sanctions-list provider over the network is not where the latency budget goes.
ISO 42001’s Annex A.9 escalation paths and EU Article 14’s two-person verification want approval inside the flow, not beside it: a named human, at a named point, before the account clears, with the approval captured as part of the record instead of living in someone’s email.
Underneath all of it sits a demand nobody writes down explicitly, but every one of these frameworks assumes: honesty about what can and cannot be undone. A sanctions hit that gets escalated and then reversed is recoverable. A Suspicious Activity Report, once filed, cannot be unfiled because the case-management step downstream turned out to be wrong. A governance layer that treats those two outcomes the same will eventually promise a reversal it cannot deliver. The honest version says which steps are compensable and which are not, before anything runs, not during the incident review.
That is the shape of the missing layer: a declared, allow-listed workflow instead of an improvised stream; approval as a gate in the flow instead of a filter beside it; a tamper-evident execution record as a first-class resource instead of a logging byproduct; and honest, declared compensation instead of a “rollback” promise that quietly doesn’t apply to half the steps.
This layer is being born now
None of the four existing layers will become this by accretion. A better dashboard doesn’t turn into a declared, reviewable workflow. A stricter filter doesn’t turn improvisation into a fixed sequence. It is a different kind of artifact, and right now almost nobody is building it as a first-class thing.
The signals that it is starting are real, though. IMDA’s whitelisting language is the first time a government framework has asked for a closed catalog of agent actions rather than a monitored open one. “Auditable Agents” and “Responsible Agentic AI Requires Explicit Provenance” (arXiv 2605.17169) both argue, independently, that provenance and tamper-evident execution records are the necessary condition for calling agentic AI accountable at all. And the IAPP’s own taxonomy, by leaving the enforcement segment out entirely, has drawn the empty square on the map for anyone paying attention.
I have been exploring what this layer looks like as a declared, versioned artifact through Actum (open source): a composition of API calls, its approval gates, and its per-step undo semantics, all fixed and reviewable before a single call goes out. That is a disclosure, not a pitch. The point of this piece is the shape of the gap, not the project.
If you are evaluating or building anything in this space, ask which of the four layers each tool occupies. Then ask the question none of them answers on its own: what, specifically, is the agent structurally prevented from doing, versus merely observed or filtered after the fact? If the honest answer is “we’d notice if it did the wrong thing,” you have very good documentation of improvisation. That is not control, and in a regulated workflow, the difference is the whole enchilada.
