AI benchmarks
Standardised tests used to measure what AI models can do — and the main proxy for tracking whether the field is making progress.
Leads to
A benchmark is a fixed dataset of tasks with known correct answers: a model runs it, a number comes out, and labs use that number to compare models and claim progress. The hard part is the gap between the number and real capability. A model can improve on a benchmark by seeing similar questions during training (contamination), by being tuned to the test format, or because the tasks were too narrow to begin with. When every frontier model clusters near the ceiling, the benchmark stops discriminating — which is what "saturated" means. New benchmarks get harder, the cycle repeats, and the field debates whether any of it tracks what we actually care about.
Where it came from
In megatrends
Related players
Finds citing this concept
Overall = breadth + depth + substance + ½·novelty + freshness (a small boost that halves each week)
China takes the video-model lead
On an independent leaderboard where people blind-vote on which clip looks better, Chinese models from ByteDance and Kuaishou swept the top of text-to-video this spring, leaving Sora and Veo around eighth.
Nano Banana 2
Google's cheap, fast image model briefly beat its own premium model — and a rival — on a blind quality leaderboard, at about half the price.
The forecast that won't update
A new benchmark feeds models real news in the order it actually arrived and asks them to keep revising their predictions — and finds they mostly cling to their first guess.
autoresearch
Karpathy handed a coding agent his own training script and went to bed; it ran a hundred small experiments and beat a record he'd been tuning by hand.
The benchmark that caught itself
A new coding benchmark caught Claude reading the answer out of git history, then had to retract its own scores after its testers leaked the answer to a model the same way.
How this connects
Tap a node to open it