Determinism is All You Need
It’s July 4, 1997, and Mars Pathfinder has just landed on Mars. More than $250 million has gone into the mission. Then the spacecraft starts resetting itself.
People were understandably alarmed. The press reported that data was being lost. In truth, data collection was delayed by about a day. For a mission with a limited lifetime, any delay was a critical concern. Engineers worked around the clock on a fix, overcoming the initial scare. This case is often presented as a classic example of a mission-critical system where determinism broke down.
Except, it isn’t. And the reasons it isn’t tell us a lot about what determinism should actually mean for AI.
What really happened on Mars
Pathfinder’s computer ran a real-time operating system, VxWorks, with tasks assigned fixed priorities. The problem was a priority inversion. A low-priority task held a shared resource that a high-priority task needed. Meanwhile, busier medium-priority work kept the low-priority task from finishing and releasing it. The high-priority task missed its deadline, and a watchdog reset the whole system.
Why was the medium-priority work so busy? The communications system was handling far more data than the engineers had anticipated.
A setting in the operating system, priority inheritance, would have prevented the inversion. It was turned off because it had a performance cost. That was a deliberate design tradeoff.
The resets had even shown up a couple of times in testing on Earth. They were shrugged off as glitches. Why? Because of time. Not just the system’s timing, but the overall Pathfinder development schedule. Anyone who has worked on large aerospace or systems engineering programs knows what gets valued above almost everything else…
Cost and schedule.
And usually schedule even more so than cost. To prevent disruptions to the program schedule, those “glitches” were not investiaged.
Why it wasn’t a determinism problem
Determinism means input A always produces output B, 100% of the time, no matter what.
Pathfinder met that definition. Given the same conditions, it reset every time. The system functioned exactly as it was designed and how all performance parameters were configured. The problem wasn’t the outputs.
It was the inputs.
The full range of conditions the spacecraft would face on Mars were never modeled.
You can argue that’s impossible to do. If we could test on Mars, we’d already be there. But the cost and schedule pressure crept in here too, shaping which scenarios got tested and which design decisions got made. The system performed as designed, in a very repeatable way. It just didn’t produce the outcome that was wanted.
This is really a story about two concepts from the systems engineering V that engineers often treat as the same thing:
- Verification asks whether you built the system you designed. You can check it against the requirements and architecture.
- Validation asks whether the system achieves the outcomes you actually wanted. That takes real-world testing, and for Pathfinder, the real world was Mars.
Pathfinder passed verification and stumbled on validation.
A lesson about trusting AI
Why does an article about determinism begin with a story that isn’t about determinism? While researching determinism case studies, Claude Opus 5.5 kept insisting Pathfinder was a great example of a determinism failure. Only after substantial pushback did Claude finally agree. Pathfinder wasn’t about determinism at all. Well, in the traditional sense. In the sense of AI agents? More on that in a moment…
The lesson here is be very careful with what AI tells you. It doesn’t matter how smart it seems. It can be confidently wrong, requiring multiple challenges to admit its error.
Where determinism actually breaks: LLM inference
In 2025, Thinking Machines Lab published Defeating Nondeterminism in LLM Inference. It deserved far more attention than it got.
They start from the same definition: same input, same output, always. And they point out something that sounds strange at first. A mathematical expression evaluates the same way every time, until a computer evaluates it.
Deterministic in theory, not in practice
In math, the order you add numbers in doesn’t matter. On real hardware, it can. Computers store numbers with limited precision, so when very large and very small numbers are added together, rounding creeps in, and the result can depend on the order of the additions.
Most of the time this doesn’t matter. When you’re working with numbers on the order of 1020, the last few digits rarely change anything. But an LLM performs billions of these calculations for every token it generates. A tiny difference can change which token comes out on top, and once one token changes, everything after it is generated from a different starting point.
This is why most software is inherently nondeterministic. Performance tradeoffs are more important that strict determinism. To achieve that performance at the hardware level, those tradeoffs have consequences. Usually they’re harmless. For an LLM, they aren’t.
1,000 runs, 80 answers
Thinking Machines sent the same prompt to Qwen3–235B 1,000 times, asking about the physicist Richard Feynman. Critically, they set the temperature to zero. In theory, that means the model always picks the single most likely next token, so the same prompt should give the same answer every time.
It didn’t.
They got 80 different completions. Every run was identical for the first 102 tokens. At token 103, they started to diverge.
The cause is a performance tradeoff
Sound familiar? GPUs split up and batch work across their cores differently depending on how much work arrives at once, and that depends on how busy the server is. For performance reasons, numbers get added in different orders. In pure math, that’s irrelevant. With limited precision and rounding happening at different points, it produces slightly different results.
Thinking Machines showed this can be fixed by changing how the work is batched, so it’s always done in the same order. But there is a performance tradeoff to strict ordering. In their tests, the deterministic version took substantially longer to serve the same requests, more than double in its first, unoptimized form.
Design decisions at the hardware and software layers have potentially massive effects at the application layer. When the same input, run 1,000 times, gives you 80 outputs instead of one, that is nondeterminism.
Theress an odd wrinkle here. No frontier model lab recommends setting the temperature to zero. Many providers now recommend temperatures around 0.7, 1.0, or even higher. Most models seem to perform much more poorly at a temperature of zero. But why? It’s a fair question. Why does nondeterministic behavior seem to be necessary to produce “good” LLM responses?
A real-world example: the Air Canada chatbot
In November 2022, Jake Moffatt’s grandmother died. He had heard Air Canada offered reduced bereavement fares, but couldn’t find the policy, so he asked the airline’s website chatbot. It told him he could book now and claim the bereavement rate within 90 days.
Get TrustGraph’s stories in your inbox
Join Medium for free to get updates from this writer.
He booked. When he later submitted his claim for the bereavement fair, his claim was denied. The chatbot’s answer contradicted Air Canada’s actual policy. According to the policy page, the airline was right, and the chatbot was wrong.
It was only a few hundred dollars, but Jake didn’t let it go. The case went to British Columbia’s Civil Resolution Tribunal, and the decision, issued in February 2024, includes some remarkable findings. First, the tribunal found that Air Canada owed him a duty of care. Then it took on the airline’s defense directly:
“Air Canada argues it cannot be held liable for information provided by one of its agents, servants, or representatives — including a chatbot. It does not explain why it believes that is the case. In effect, Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission. While a chatbot has an interactive component, it is still just a part of Air Canada’s website. It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.”
— Moffatt v. Air Canada, 2024 BCCRT 149
The tribunal also found that Air Canada did not take reasonable care to ensure its chatbot was accurate, and that customers shouldn’t have to double-check information found in one part of a company’s website against another. In other words, it isn’t the customer’s job to verify what the chatbot says.
This case predates the generative AI boom, but the reasoning lands squarely on today’s questions about liability, inconsistency, and automated systems, like AI agents, that get things wrong.
What determinism means for products
Is the Air Canada chatbot a determinism problem? Not in the input-A-equals-output-B sense. Every customer’s question is different. This is the Pathfinder problem all over again. We can’t realistically model every possible input. And if we can’t know all the inputs, we can’t know all the outputs.
So for a product, especially an autonomous one like an AI agent, determinism has to come down to outcomes. Are the results of its actions good, bad, or somewhere in between, where a human needs to intervene? That’s the question that matters, and it leads straight to governance.
Duty of care, due care and risk
That’s where risk comes in. There is a risk framework that is badly underrated, the Duty of Care Risk Analysis Standard (DoCRA). It defines two terms that map directly onto the Air Canada ruling:
- Duty of care: the responsibility of one party to prevent harm to others. Air Canada had a duty not to mislead its customer.
- Due care: a degree of protection that a reasonable person applies to protect others from harm. The tribunal found Air Canada didn’t take reasonable care to make its chatbot accurate. It didn’t exercise due care.
This is going to come up constantly with AI, and it already has. Unfortunately, the courts haven’t settled much yet. One of the highest-profile cases is Amazon’s suit against Perplexity over its Comet browser, whose AI agent can shop on Amazon on a user’s behalf. A central question is who is responsible for what an agent does. An injunction was granted, then vacated on appeal, and the case continues. The courts seem to be leaning toward holding the operator of an agent responsible for the outcomes. Even if a company deploys an agent someone else built, the company running it has the duty of care.
Risk management doesn’t have to stop you building
There are many misunderandings aroudn risk management. Some believe that by simply performing risk analysis, they will then be handcuffed from pursuing high risk, high reward outcomes. That’s simply not how risk management works.
You can accept high risk, and justify it, especially when the return is high. For a startup, the whole venture is high-risk, and some decisions are existential. Accepting a risk the company can’t survive without is often entirely reasonable.
DoCRA addresses this problem directly. Its principle on safeguards is that they must not be more burdensome than the risks they protect against. It offers two ways to test that: compare the total burden of your safeguards against the total risk, or compare each option’s added burden against the risk it removes. The bar is not more burdensome, as opposed to far outweighs.
There’s a legitimate tradeoff. If preventing a harm would cost more than the harm itself, it can be reasonable to not take action. What matters is showing that you made the judgment, did the analysis and documented why. That analysis, reasoning, and documentation is your due care.
Likelihood and impact aren’t enough
Traditionally, we measure risk as the likelihood of an event times its consequence, often scored high, medium or low. That works when you’re weighing ten risk scenarios.
It breaks down when there are thousands, or millions. In a complex agentic system, every step involves a decision whether it’s which data to use, which tool to call, or which output to prioritize. Automate all of that, and the number of decisions spirals out of control. Governing agents needs more factors than just likelihood and consequence. For example:
- Time: Is the information behind the risk analysis still valid, or has it changed? Almost every risk framework misses this.
- Uncertainty: Do we have enough information to reach a reasonable conclusion? We ask whether results are statistically significant in mathematics; we rarely ask it about risk.
- Outcome value: If outcomes can be good or bad, we have to be able to put a value on them. It’s hard, but it’s doable; people model things as intangible as a company’s brand.
- Scope: How far does an action reach, and who does it affect?
- Authority: Who or what has the authority to take this action at all?
What determinism will look like for AI agents
Determinism for autonomous systems isn’t going to mean input A produces output B every time. We won’t be able to run an agent 1,000 times and get the identical result all 1,000 runs.
What we should be able to expect is once we define the outcomes we want, the outcomes we don’t want, and the outcomes we’re unsure about, then every single time, under those parameters, the system will approve the action, deny it or pause and hand it to a human to decide.
That’s what determinism will look like for agentic systems. And what are we really talking about?
Governance.
It’s also exactly what we’re building into TrustGraph. It’s been a cornerstone of our vision from the beginning, and it matters more now than ever before. People are asking not just how to make their data more accurate, but how to guarantee an agent won’t do something they don’t want, and that if it tries, it stops and asks someone to approve or deny it.
We’ve all seen what happens when you tell an LLM to stop and it carries on anyway, or research showing models that know they’re doing something they were explicitly told not to do. Guardrails may be able to prevent a LLM from flying off a cliff, but it won’t stop them from running headfirst into an oncoming truck. That’s why the decision control has to sit in a system outside the LLM, one that is itself deterministic.
Govern your agents today with TrustGraph.
Sources and further reading
- Mars Pathfinder: Priority Inversion Problem
- Defeating Nondeterminism in LLM Inference, Thinking Machines Lab
- Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia
- Duty of Care Risk Analysis (DoCRA) Standard v1.1
- Amazon v. Perplexity: Ninth Circuit ruling on AI agents, National Law Review
- TrustGraph · TrustGraph on GitHub
