Member-only story
How to measure AI
AI benchmarking is like competing who has bigger Schwartz. It keeps the competition going, but outside of a small group of people, the numbers have almost no meaning.
To illustrate my point, let’s go to BenchLLM. We can see that Gemini 4 Carbon has a score of 88.79. The Agentic Score is not known right now, though the Coding score is at 81. Coming in at a close second is Claude Opus 5.5 with a score of 86.4, an Agentic score of 90 and a Coding score of 83. Third is GPT Astra with 86.04, 76 and 75, respectively.
Of course, that’s what BenchLLM people think. People at Artificial Analysis rank Gemini 4 Carbon (high) at 60, Claude Opus 5.5 (max) at 59 and Gemini 4 Carbon (medium) at 58. (These scores are based on the Intelligence Index.)
To get more numbers, you can visit LiveBench, Arena Leaderboard or Scale AI.
Everyone has their own methods and is getting their own numbers. But they are all Schwartz competition numbers — like ‘DistressBench’ which gives you the quality of crisis support that the particular model delivers.
This is the problem with benchmarks. If I take you to the year 1989, we were comparing 286, 386 and 486. While 286 was between 6 MHz to 25 MHz, the 386 was 12 MHz — 40 MHz, and the 486 was 16 MHz — 160 MHz.
