Mentatcurated
Artificial Intelligence medium · first-party

Pelicanmaxxing

Someone finally measured whether AI labs secretly train for the famous pelican-on-a-bicycle test, and the answer is no.

For two years the internet's favorite way to size up a new AI model has been to ask it for a drawing of a pelican riding a bicycle. The prompt is deliberately silly, which is the point: a lab supposedly cannot study for a test it does not see coming. But as the pelican became a ritual, a suspicion grew alongside it, half-joking on Hacker News and Twitter, that the big labs had quietly started optimizing for it. Nobody had checked.

Dylan Castillo did. He built a grid of 8 animals crossed with 6 vehicles, had 7 frontier models each draw all 48 combinations, and used a simple statistical adjustment to ask one question: does any lab draw the pelican-bicycle cell better than that same lab's general skill at pelicans and at bicycles would predict? If a lab were cramming for the famous prompt, this specific square would light up. None did. The strongest hint, one Chinese model running slightly hot on the pelican cell, washed out as statistical noise.

The tell is that the pelican is not even good. Among the eight animals it ranked sixth; the bicycle came second-to-last of the vehicles; the pelican-on-a-bicycle pairing landed 42nd of 48. The single most obsessed-over benchmark in AI turns out to be one of the models' weaker subjects, the opposite of what gaming would produce. The whole investigation cost about eighty dollars in API credits.

The honest limit is sharper than the headline. Castillo can only clear the labs of pelican-SPECIFIC tuning; a lab that simply got good at SVG drawing in general would rise on every cell at once and look identical to a lab that is just good. So the pelican is safe as a probe of raw skill, and unprovable either way as a probe of intentions.

The lenses

Novelty 3
Impact · breadth 2
Impact · depth 2
Actionable 2
Delight 4
Substance 4
Hype 3

The facts

Open dylancastillo.co →

How this connects

Tap a node to open it

How this connects