Token Races
I have done my share of complaining about how fast Claude goes through a week’s tokens (Steve Yegge, who burns like an order of magnitude more than I do, has similarly complained). But this weekend I accidentally did an A/B test of Claude vs. Codex’s $200 tier accounts, and hate on Anthropic all you want, but you get way more gas for your dollars out of Claude. Like 4 to 12 times as much, depending on how you measure it.
This last Friday, I added Codex support to yoyo (https://github.com/swarmgraph/yoyodyne), my pet software factory (there are many like it, but this one is mine). By the time I finished, Claude (the $200 Max 20x plan) had run out of tokens for the week, so I let Codex (the $200 ChatGPT Pro plan) run everything. It burned through all its tokens in less than a day — turns out a bug had it defaulting to extra high effort and it was reading in a bunch of instructions that it shouldn’t. Fixing all that took one of my Codex resets (every so often Codex hands you a fresh window of tokens), then it ran for a few more hours on that before it needed another reset. With all that cleaned up, each role was running Codex’s equivalent of its Claude model, at similar effort levels:
- Architect and development manager: Fable at high effort on Claude; Astra at high on Codex.
- Developers and reviewers: Opus at medium effort on Claude; Sol 6.1 at high on Codex.
I assumed it would give me about 3 days’ worth of work, just like Claude would.
But no… On that last, clean window, Codex ran my software factory for about a day and produced about 18 PRs. Compare this to Claude, which had most recently run it for about three and a half days and produced about 84 PRs. Claude’s weekly allowance carried about 9 times as much input, 12 times as much output, and 5x more PRs for the same $200 / month.
The numbers
What one week’s allowance bought:
Uncached input is what the model had to read in full rather than from its prompt cache. About 95% of Claude’s input came from the cache, against 89% of Codex’s. If the plans charge cached input at a discount, as the APIs do, better caching explains some of the gap. Counting only uncached input, Claude’s allowance still carried about 4 times as much.
Get mason bryant’s stories in your inbox
Join Medium for free to get updates from this writer.
Yes, PRs are a poor measure of dev productivity, but they’re the best we have. Per day the counts are close (ish), 24 for Claude and 18 for Codex, but the totals for a single week’s token window are super different.
PR size
Claude’s PRs are longer: 890 lines per PR against 490 for Codex. Without comments, they’re almost the same size (average of 553 for Claude, 515 for Codex). Output tokens point the same way: about 200,000 per PR for Claude against about 78,000 for Codex.
That matches my experience in conversation, where Claude is much more of a yapper than ChatGPT. Spot-checking, some of the comments are useful, but many are Claude writing the history of the bug or failure it’s fixing into the code. I see the same thing when I use Claude for interactive coding.
Caveats
To be clear — this isn’t some kind of scientific study:
- The work items were different.
- The days were different.
- The sample size is small.
I’ll do this again next week once each account’s weekly window resets, but for now I’m operating on the assumption that I’m just going to get more work out of Claude.
Have any of y’all seen anything that agrees or disagrees with this?
