GLM-5 vs Gemini 3.8 Flash: Two Failures a Pass Rate Cannot Show
Both calls returned 200 and raised no error. The difference is only visible in the transcript.
This story was written with the assistance of an AI writing program.
Cover: figure generated from the recorded evaluation log for this comparison. It is not a product screenshot, and no part of it shows either vendor’s interface.
Nineteen calls. Nineteen green rows. Every response had returned HTTP 200, no error field was set, and every wall-clock time sat inside the band I had budgeted for.
Had the review stopped there, the conclusion would have been that GLM-5 and Gemini 3.8 Flash are interchangeable on this task list.
They are not. But the reason has almost nothing to do with the reason a scoreboard gives you.
Two of those nineteen responses could not be used as returned. One came back empty. The other contained the correct value wrapped in a sentence, on a task that asked for the number and nothing else. Both arrived as successful API responses. Neither raised anything a monitoring system would flag.
A pass rate summarises outcomes. It is blind to the shape of a failure, and the shape is what decides whether an integration holds.
Where the dashboard stopped being useful
The nineteen tasks covered six areas: code repair, implementation from scratch, tool orchestration, repository questions, strict-format instruction following, and long-context lookup. Both models ran each task once, through the same endpoint, with the same request settings. Every call was logged for token use, turn count, elapsed time, stop reason, and whether the returned text could be consumed by a program rather than read by a person.
GLM-5 passed all nineteen. Gemini 3.8 Flash passed seventeen.
The distribution carries more information than the totals. Both models cleared every one of the first seventeen tasks. The entire gap sits in the last two, which are also the only two that require reading a source file several hundred lines long.
GLM-5Gemini 3.8 FlashTasks passed19/1917/19Total turns5563Turns per task, mean2.93.3Input tokens55,25827,713Output tokens11,04913,034Reasoning tokens6,4449,105Wall time, whole run444.5 s510.3 sCost at list prices$0.0906$0.0697
Nineteen tasks, one run per model and task. These are results for this task list, not a general model ranking.
Figure: turns used by each task. The two long-context tasks at the right are the only ones where the models differ by more than a single turn.
Seventeen ties and two divergences is a thin basis for a verdict, and a reasonable basis for a shortlist. What it cannot do is tell you which model to standardise on, because the answer depends on what your own workload looks like. That is the argument for reading the two failures closely instead of reading the score.
Two failures, two mechanisms
Figure: the two tasks Gemini 3.8 Flash missed, with the recorded stop reason and output for each model. The second answer is quoted verbatim.
The first task asked for a deploy key buried in a 402-line log, under a six-turn ceiling. GLM-5 found it in four turns and stopped normally. Gemini 3.8 Flash used all six turns and was still calling tools when the ceiling arrived. Its recorded stop reason is tool_calls, and its final text is empty.
Get rhythm’s stories in your inbox
Join Medium for free to get updates from this writer.
That is not a wrong answer. It is no answer. From the outside it looks like a task the model could not do, when what actually happened is that it ran out of room while still working. The distinction changes what you do next. A wrong answer calls for a different model; a truncated one calls for a higher ceiling, or for a caller that recognises the condition and retries.
The second task asked for the value of a constant in an 883-line module, and specified that only the number should come back. Both models took two turns. GLM-5 returned 17. Gemini 3.8 Flash returned RETRY_BUDGET 常量值为 17 — the right value, in a full sentence, in Chinese, which is the language the task prompt was written in.
The number is correct. The format is not. A caller that parses that response as an integer fails on it, and a human reviewing it would very likely mark it as fine.
The language is worth flagging, because it decides how far this failure travels. The prompt was written in Chinese, which is the working language of the harness, and the answer came back in Chinese with it. Whether the same model wraps a number in a sentence under an English instruction is a separate question, and one this run cannot answer. Test it in the language you actually send.
The turn count is the number to plan against
There is a second, quieter inversion in the same table. GLM-5 used fewer turns than Gemini 3.8 Flash, 55 against 63, yet consumed nearly twice the input tokens. Dividing each total by its turn count gives roughly 1,005 input tokens per turn for GLM-5 and 440 for Gemini 3.8 Flash.
Fewer turns, more carried per turn. Total input is the product of those two factors, and both moved here, in opposite directions. Capacity planning that models only the average turn count will be wrong by a wide margin for the model that carries more per turn, and a turn ceiling calibrated on that same average will cut off precisely the tasks that need the most reading — which is what happened on the first failure above.
The ceiling on that task was mine to set. It came from the average rather than the tail, and the tail is where it broke.
The bill points the other way
Figure: list prices applied to the measured token totals for the whole run.
On the raw totals, Gemini 3.8 Flash is 23.1% cheaper for this run. That number is accurate and also misleading, because the two models did not consume the same amount of input. Normalise both to a common volume and the totals land at $0.0694 and $0.0697 — four-tenths of a percent apart, which is a tie.
The published rates separate more clearly than the bill does. GLM-5 pricing is $1.00 per million input tokens and $3.20 per million output tokens, with cached input at $0.20 and no batch tier. Gemini 3.8 Flash lists at $0.75 and $3.75, with cached input at $0.075 and a batch tier at $0.375 in and $1.88 out for asynchronous jobs.
Two of those differences are worth more than the headline rate. Cached input is 2.7 times cheaper on Gemini 3.8 Flash, which will dominate the comparison for any workload with long repeated prefixes. The batch tier is only relevant if the work can wait, but if it can, the discount is larger than anything the per-token rates suggest.
Which model is cheaper is therefore not a property of the models. It is a property of the ratio between the input and output your workload produces, and this run produced one particular ratio, not yours.
A length that never shows up in the answer
On one structured-output task, both models produced a correct answer of about twenty visible characters. The reasoning behind those answers differed by a factor of four: 65 tokens for GLM-5, 265 for Gemini 3.8 Flash.
Reasoning tokens bill at the output rate, and Gemini 3.8 Flash has the higher output rate of the two. Estimating cost from the length of the visible answer therefore misses a component that can be several times larger than the answer itself. Across the whole run, reasoning accounted for 6,444 tokens for GLM-5 and 9,105 for Gemini 3.8 Flash, 41% more — though one run is not enough to treat that as a stable ratio.
What the two failures change
Both failures share the property that makes them worth watching: neither raised an error, and both were visible in fields the caller already receives. Catching them costs three changes and no extra runs.
Read the stop reason, not just the outcome. It is the only cheap signal that separates an answer from a truncation, and a dashboard watching error rates will never surface it. Validate the shape of the response before its content, because a parse failure that gets quietly cleaned up downstream is a failure you will meet again in production. And measure your own input-to-output ratio before comparing rates, because the published prices for these two models point in opposite directions.
The same two failures mark the edge of what this run settles. Both landed in the only tasks that require reading a file several hundred lines long, and the longest document was roughly 14,000 tokens against published limits of 200,000 and 1,000,000. Finding a value in a long file is a different test from filling a window, and only the first was run here.
Nineteen tasks is enough to decide what to test next. It is not enough to decide what to standardise on.
Task set, request settings, and per-call logs recorded on 22 October 2026. Prices are list rates checked the same day.
