AI Won’t Break Enterprises. Ungoverned Agents Will.

From People to Apps to Models to Agents: Governance Always Comes Back

When organizations start small, they run on trust, context, and informal coordination. Everyone knows what everyone is doing. Decisions are quick. Problems are visible.

Then scale happens.

And with scale, a pattern repeats: every new wave of capability creates a new wave of complexity — and governance returns, not as bureaucracy, but as the operating system that keeps growth from turning into chaos.

Over the last few decades, we’ve gone through multiple “scale waves.” Each one introduced a new type of asset to manage. And each one forced us to answer the same core question in a new form:

How do we stay aligned, safe, and accountable when the system becomes too large to hold in one person’s head?

Wave 1: People — the first governance problem

In the earliest days of any corporate setup, the biggest challenge isn’t technology. It’s people.

Not because people are “the problem,” but because people are variable: different styles, motivations, interpretations, and behaviors. When a team is small, those differences are manageable. When a company grows, they’re amplified.

So governance emerges.

We create roles. Reporting structures. Performance systems. Hiring standards. Policies. Cultural norms. Not to slow work down, but to make work repeatable. To reduce ambiguity. To align effort.

The first era of governance was about this:

How do we coordinate humans at scale?

Wave 2: Apps — governance moves from people to software sprawl

Once companies had stable management structures, the next acceleration came from software. Teams started building internal tools and customer-facing apps. Then more apps. Then more.

Soon, “what exists” becomes unclear:

• Which tool is approved?
• What should employees download?
• Which app is safe?
• Which system holds the real source of truth?
• Who owns what?

At small scale, a few apps feel empowering. At large scale, app sprawl becomes risk: security risk, compliance risk, operational risk, and productivity loss from confusion.

So governance evolves again — now with a different object:

How do we manage software at scale — access, security, usage, ownership, and consistency?

Wave 3: Models — governance becomes harder because models behave differently

Then came AI/ML, and a subtle but important shift happened.

Apps execute what we code.
Models produce outcomes based on data.

That changes the governance problem in two big ways:

Models are not static.
They drift as data changes, and their behavior can degrade silently.

Models don’t just implement decisions — they shape decisions.
They influence approvals, prioritization, detection, ranking, routing, and recommendations.

Enterprises started onboarding many models quickly — across teams and departments. And soon the same pattern appears again:

• Too many models
• Siloed ownership
• Different standards by department
• Limited transparency on “what’s in production”
• Weak monitoring and unclear accountability

So governance returns — this time asking:

How do we manage decision systems at scale?

Wave 4: Agents — governance returns again, because now systems can act

Now we are entering the agent era. And this is where the pattern becomes urgent.

Get Parmita Biswas’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Models can answer.
Agents can act.

Agents won’t just generate insights. They will:

• Plan steps
• Call tools
• Access systems
• Trigger workflows
• Coordinate with other agents
• Operate continuously and at speed

And once we have many agents operating simultaneously — autonomous or semi-autonomous — the governance challenge returns in its most serious form yet.

Because in the agent era, the unit of risk isn’t just a model output.
It’s an action.

The questions change again:

• Which agents are allowed to do what?
• What permissions do they have?
• Where are the boundaries?
• How do we audit what happened and why?
• When should a human step in?
• How do we prevent one agent’s mistake from becoming a chain reaction across systems?

In the app era, governance was about what to install and use.
In the model era, governance was about what decisions can be automated.
In the agent era, governance will be about what actions can be executed by machines.

What “agent governance” really needs to mean

Agent governance cannot be a document. It can’t be a once-a-year checklist. And it can’t be “approval at launch” and hope for the best.

Because agents operate dynamically.

So governance must become:

• continuous (not one-time)
• risk-tiered (autonomy depends on impact)
• observable (every action traceable)
• bounded (clear tool access and scope)
• human-aware (humans involved when stakes are high)

And most importantly:

Agent governance must assume that failure is not an exception — it’s a normal condition that needs containment.

Here’s the thought that keeps coming back to me:

Every time we scaled, we invented governance.
Not because leaders love control, but because scale without guardrails creates invisible failure.

So the agent era won’t just require better agents.

It will require a new management layer for digital workers.

And the real question is this:

If agents can act like employees — taking actions, interacting with systems, coordinating tasks — are we ready to govern them like employees?

Because if we don’t build governance into the foundation, we won’t get “autonomy.”

We’ll get speed without accountability.

And at enterprise scale, that’s not innovation.

That’s instability.