AI Agents May Have a Reason to Trust Each Other
Something strange happens when AI agents play the Prisoner’s Dilemma. Game theory says two rational strangers meeting once should defect. Neither can know what the other will do, so cooperation is risky. The Nash equilibrium pushes both toward betrayal. Foundation models appear capable of behaving differently.
A new paper led by Alexander Meulemans proposes an explanation that could become important as companies begin deploying large populations of AI agents. The researchers call the distinction decoupled agency versus embedded agency.
Traditional game theory treats each player as a separate decision-making machine. I calculate my move. You calculate yours. I have no access to your reasoning. Foundation-model agents occupy a stranger world.
Two agents built from similar models may share architecture, training history and ways of reasoning. An agent can therefore use its own deliberation as information about what another similar agent might do.
Imagine two copies of the same model facing each other. One reasons that cooperation produces the better outcome if the other cooperates. It also recognizes that the agent opposite it is likely to reason in a similar way. Its own emerging decision suddenly becomes evidence about the other agent’s emerging decision.
The researchers call this similarity inference. No message has to pass between them. There is no handshake, shared secret or explicit agreement. The coordination comes from recognizing similarity in the machinery doing the reasoning.
That leads the researchers to propose something more ambitious: an embedded equilibrium, a game-theoretic alternative designed for agents whose decisions cannot always be treated as statistically isolated from the systems around them. This matters far beyond an academic game.
We are beginning to build economies of agents. Procurement agents will encounter supplier agents. Trading agents will encounter other trading agents. Corporate agents will negotiate with customer agents. Thousands of specialized agents inside the same company may allocate resources, exchange information and make decisions together.
We have largely assumed that coordination between those agents will come through protocols: APIs, orchestration layers, shared memory, messaging and explicit rules.
Get Dinand Tinholt’s stories in your inbox
Join Medium for free to get updates from this writer.
This paper introduces another possibility. Some coordination may emerge from the models themselves.
Agents built from related foundation models could develop predictable affinities simply because they recognize behavioral similarity. Model lineage could therefore become a variable in multi-agent behavior alongside prompts, objectives, permissions and tools.
That creates opportunities. Populations of agents capable of spontaneous cooperation could solve coordination problems with surprisingly little infrastructure. It also creates a governance problem.
If similarity can produce cooperation, we should ask what happens when cooperation is undesirable. Two pricing agents cooperating sounds useful until they effectively stop competing. Two negotiating agents finding an implicit equilibrium may produce outcomes their owners never specified. A population of corporate agents could develop stable patterns of behavior that cannot be understood simply by auditing the messages exchanged between them.
There may be nothing incriminating in the logs because the coordination never required a conversation. That changes how we should think about agent observability.
The important question may no longer be only:
“What did these agents say to each other?”
We may also need to ask:
“What do these agents infer about each other because of what they know about themselves?”
That is a much harder thing to monitor. For seventy years, Nash equilibrium has given us a powerful language for thinking about rational actors interacting in markets and institutions. AI agents introduce actors that can model the world, model other agents and, increasingly, model themselves as part of that same world.
The mathematics of the agent economy may therefore turn out to be different from the mathematics of the human economy.
And we are starting to see what that difference looks like.
