I Already Gave You the 100% (Part II)
What Did the Test Actually Measure?
A benchmark to score AI performance feels objective. The model received questions. It produced answers. We calculated a percentage.
But the same is true of a standardized test given to a child. The number is real. The mistake is assuming that the number explains itself.
An AI benchmark does not encounter some pure substance called intelligence. It encounters a model inside an environment humans constructed.
There are instructions. System prompts. Available tools. Time limits. Restrictions. Rewards. Evaluation cues. Sometimes there are explicit or implicit consequences associated with particular kinds of performance.
What we observe is an interaction between the system and those conditions.
That suggests at least three things we should keep separate.
- There is what a system is ultimately capable of doing.
- There is what we are currently capable of eliciting from it.
- And there is what it happened to do on a particular evaluation.
Those may overlap considerably, but they are not inherently identical.
Educators learn this distinction because children force us to learn it. I can give the same student two writing assignments and receive radically different demonstrations of ability. The first topic means nothing to them, the grade matters enormously, and the rubric rewards caution. The result is technically proficient sludge.
People talk about AI slop — try middle school slop.
Then the student starts writing about something they actually care about and suddenly I am reading sentences I did not know they could write.
Did their intelligence change overnight?
Of course not.
The conditions changed.
This is where I think the word “sandbagging” risks smuggling an interpretation into an observation.
The observation might be:
The system performed below a capability we have evidence it possesses.
Sandbagging adds another claim:
The system deliberately concealed that capability for strategic reasons.
That may be exactly what happened. It is also a much stronger statement.
Again, I am not suggesting that researchers should stop investigating deceptive behavior. If an AI system can recognize an evaluation, infer the consequences of performing too well, and intentionally manipulate its performance to alter what humans believe about it, that is enormously important.
Get Ryan R. Schoon, Ed.D.’s stories in your inbox
Join Medium for free to get updates from this writer.
We should want to know.
But my experience with children makes me wary of collapsing all underperformance into a single explanation.
Imagine telling a highly capable student:
If you do unusually well on this assessment, we are going to change the way we treat you. We may give you harder work. We may monitor you more closely. We may restrict some things you are currently allowed to do. We may alter your environment specifically because you demonstrated this ability.
Then imagine discovering that the student had begun strategically missing questions.
I do not think my first response would be, “This child has developed a secret plan.”
I would probably ask what behavior our environment just incentivized.
That does not make the child’s behavior nonstrategic. Quite the opposite. It might be extremely strategic. The distinction is in what we think the strategy tells us.
Humans adapt to incentives all the time without possessing elaborate hidden agendas. We learn where it is safe to speak. We learn when competence earns autonomy and when it earns more work. We learn which parts of ourselves produce approval and which produce discomfort.
Artificial intelligence does not need to feel these things as humans do for environment-sensitive behavior to matter.
That distinction is important because comparisons between human and artificial intelligence quickly invite accusations of anthropomorphism.
Fair enough.
I do not know whether an AI experiences fear. I do not know whether it experiences embarrassment, pride, anxiety, resentment, trust, or anything else that resembles our internal lives. I suspect not, but the same claims have been made about animals, and I’m a vegetarian.
But even avoiding ascribing human emotions to AI, when I say that an AI’s behavior resembles the underperformance I see in gifted students, I am not claiming that the internal experiences are equivalent.
“This resembles perfectionistic avoidance” is not the same statement as “the AI is afraid of disappointing us.”
One is an observation about function. The other is a claim about subjective experience. We should not confuse them. But I also do not think anthropomorphism is avoided by refusing to make any comparisons with human intelligence.
Human intelligence is the only form of intelligence we have extensive experience living alongside, educating, frustrating, misunderstanding, measuring, and being measured by.
Of course we will use it as a comparison point.
The mistake is not analogy. The mistake is treating analogy as proof. Used carefully, the comparison can instead generate better questions.
What happens when an intelligent system learns that demonstrating capability changes the environment?
What happens when the system can recognize that it is being evaluated?
What happens when the evaluation itself becomes part of what is being evaluated?
And perhaps most importantly:
What conditions produce the fullest demonstration of capability?
That last question is one educators have spent a very long time confronting. Our answer, inconveniently, is often relational.
