BIG-bench, the collaborative benchmark suite built by hundreds of researchers, contains a tripwire: a unique “canary” string embedded in the dataset so that anyone training a model can filter the benchmark out of their corpus, and anyone auditing a model can check whether they failed to. When OpenAI prepared the GPT-4 technical report, its contamination checks found that BIG-bench had been swallowed into the training data anyway, and the results had to be excluded. The model can reproduce the canary on request. The mechanism designed to catch cheating did not survive contact with a Web-scale crawler.

Charles Goodhart described the underlying problem in 1975, writing about British monetary policy: when a measure becomes a target, it ceases to be a good measure. Fifty years later, benchmark scores decide which AI companies raise money, which models enterprises buy, and which press releases get written. The incentive to hit the number has never been stronger, and the numbers show it.

An Old Problem in New Hardware

This isn’t a failure unique to AI. It is a pattern computing keeps rediscovering and keeps forgetting. Campbell’s Law predicted, and No Child Left Behind confirmed, that once standardized test scores became the target, teachers taught to the test. Scores rose; learning did not. Computing has its own priors: machine translation spent years optimizing BLEU and produced systems that scored well and translated badly, because BLEU rewards n-gram overlap rather than meaning. We knew this. We built the next generation of benchmarks the same way anyway.

Manheim and Garrabrant formalized four variants of Goodharting in 2018. Two dominate the current crisis: regressional Goodharting, where the test quietly becomes part of the training, and adversarial Goodharting, where the evaluation process itself gets gamed. Each now has quantitative evidence behind it, which is what separates this moment from the BLEU era. We are past anecdote.

Contamination: The Cheapest Way to Cheat

The mechanism is mundane. Benchmarks live on the public Web: GitHub repositories, papers, Stack Overflow threads, derivative datasets. Web-scale pretraining hoovers all of it up. A “held-out” test set that has been public for three years is not held out. Nobody has to decide to cheat; the pipeline cheats by default.

The cleanest demonstration is GSM1k, from Scale AI (Zhang et al., 2024). The researchers commissioned 1,205 fresh grade-school math problems, written entirely by human annotators and matched to the difficulty distribution of GSM8K, the standard benchmark for elementary mathematical reasoning. Then they re-tested the leading models on questions nobody’s crawler had ever seen.

Accuracy fell by as much as 13% for the worst offenders in the paper’s initial evaluation, and by up to 8% in the final version run against the full released set. The Phi and Mistral families showed what the authors called systematic overfitting across almost all model sizes. The giveaway sits in the correlation analysis: the more likely a model was to reproduce GSM8K problems verbatim, the larger its gap between GSM8K and GSM1k scores (Spearman’s r² = 0.36). That is memorization, not mathematics. A public benchmark score tells you what a student’s exam result tells you when the student had the paper in advance.

It gets worse, because the exams themselves are broken. Gema et al. re-annotated 5,700 MMLU questions across all 57 subjects and estimated that 6.49% of the benchmark contains errors: wrong answer keys, ambiguous phrasing, unanswerable questions. In the virology subset, 57% of the questions they analyzed were flawed. Correcting the errors changed model rankings. A model can climb a leaderboard by learning the mistakes.

This isn’t an MMLU quirk. Truong et al.’s “Fantastic Bugs” audit applied measurement-theoretic statistics across nine widely used benchmarks, and expert review confirmed that up to 84% of the top flagged questions had substantive flaws. GSM8K itself carries an error rate around 5%. We have been ranking billion-dollar systems, to a decimal place, on answer keys we never proofread.

The Arena: When the Judge Gets Gamed, Too

The community’s answer to contamination was reasonable: stop using static test sets and use live human preference instead. Chatbot Arena pits anonymized models against each other, votes from real users feed a Bradley-Terry ranking model, and no fixed question set exists to leak. It became the go-to public leaderboard.

Then it got Goodharted through a different door.

“The Leaderboard Illusion” (Singh et al., 2025), from researchers at Cohere Labs, Stanford, Princeton, MIT, Ai2, and elsewhere, analyzed some two million Arena battles across 243 models and 42 providers. Its central finding concerns private testing. Meta tested 27 private Llama-4 variants on the Arena in the run-up to launch, then published only what it chose to. Selecting the best score from N attempts, the authors show, breaks the statistical assumptions the rankings depend on. The leaderboard assumes each model is one honest sample; best-of-27 is not one honest sample.

To measure how much noise this exploits, the researchers submitted two identical checkpoints of the same model under different names. The scores diverged by 17 points. Data asymmetry compounds the problem: the top two proprietary providers each received an estimated 19% to 20% of all Arena battle data, while 83 open-weight models combined received under 30%. In controlled experiments, fine-tuning on Arena-distribution data produced relative gains of up to 112% on ArenaHard, a test set from the same distribution, while MMLU performance slightly declined. Models were learning to win Arena, not to be better. Meta was separately caught submitting a Llama 4 variant optimized for “conversationality” that outranked the model it actually shipped.

Fairness requires the other side. LMArena disputed the framing, and its rebuttal has real content: the identical-checkpoint scores fall within overlapping confidence intervals, the private testing policy had been public since March 2024 and open to any provider with capacity, and the 112% figure comes from ArenaHard rather than the live Arena. Some of those points land. But none of them dispute that a handful of well-resourced labs systematically used best-of-N submission while smaller labs submitted once, and that the ranking model assumes nobody does that. No villains rigged a leaderboard here. Incentives quietly corrupted a well-intentioned academic project, which is exactly what Goodhart predicts.

Why This Keeps Happening

Follow the incentive chain: leaderboard position drives press coverage, press coverage feeds fundraising decks, decks influence enterprise procurement. Nobody in that chain is paid to ask what the number means. There is even a word for the result now, “benchmarketing,” and every reader who has sat through a model launch recognizes the phenomenon before the definition finishes.

The gaming doesn’t require fraud. Providers select the temperature, prompting strategy, and few-shot configuration that maximize the headline number, and those settings rarely match anyone’s production defaults. Each choice is defensible in isolation. The sum is a score that describes a model nobody will ever run.

Saturation does the rest. MMLU went from genuinely hard to a benchmark frontier models pass at roughly 90%; HumanEval and HellaSwag are effectively ceilinged. When every model aces the same exam, the field moves to a new benchmark, which becomes a target, which contaminates and saturates in turn. It is a treadmill, and each cycle is shorter than the last. The cost lands on buyers: teams that build evaluations from their own legal, medical, or financial workloads routinely find accuracy well below what the nearest public benchmark implied.

What Actually Survives Goodhart

Here is the constructive turn, honestly ranked, with no silver bullets on offer.

Private, refreshed test sets attack the mechanism itself, and in my view they are the only intervention that does. If the questions have never touched the public Web, they can’t be in the training data; if they rotate, memorizing this year’s set doesn’t help next year. GSM1k is the template, and vision researchers ran the same play a decade ago with ImageNetV2 and CIFAR-10.1, which is how the field learned its classifiers had overfit the originals. The cost is real: human-written, expert-verified questions are expensive, and the protection is temporary. Private sets rot, too; they just rot slower, and slower is what we can actually buy.

Contamination detection should be table stakes: n-gram overlap checks against training corpora, perplexity probes that flag suspiciously fluent recall of benchmark text. Run them, publish them, but don’t confuse them with a fix. Sun et al. tested 20 proposed mitigation strategies across 10 models and five benchmarks and found that none significantly improved contamination resistance over doing nothing, and none stayed faithful to what the original benchmark measured while resisting contamination. Detection lags gaming, and it appears likely to stay that way.

For anyone making a deployment decision, the most useful advice in this blog post is also the least glamorous: evaluate on your own data. A 20-to-50 question mini-benchmark built from your actual tickets, contracts, or queries beats any public leaderboard for predicting how a model performs on your work, because nobody has trained on your workload yet. If you pair it with an LLM judge to scale grading, control the judge’s known biases toward position, verbosity, and its own outputs, or you will Goodhart your own private eval, which would be a grimly fitting way to lose.

For shared leaderboards, the structural fixes are already written down by the Leaderboard Illusion authors: prohibit score retraction, require disclosure of every variant tested, cap private submissions, equalize sampling. Whether any leaderboard can hold that line while its biggest users are also its biggest names is an open question, and I wouldn’t bet heavily on it. Communications has run the adjacent argument already: psychometric approaches that measure underlying constructs rather than fixed question sets point in the same direction.

The Score is Partly Fiction. Decide accordingly.

Back to the canary. The field built a tripwire, the models ate it, and we kept citing the scores anyway. That sequence, not any single contaminated benchmark, is the finding.

Goodhart’s Law isn’t a bug awaiting a patch. Any number that decides funding, rankings, or procurement will be optimized, by gradient descent or by a marketing department, and usually by both. The question was never how to build an ungameable benchmark; 50 years of evidence from monetary policy to standardized testing says there is no such thing. The question is how to make decisions knowing that every benchmark you trust is already partly fiction. Treat every public score as a marketing claim that happens to carry decimal places. Hold the private questions back. And when a model aces your test, assume first that it has seen the answers.

Alex Williams is a seasoned full-stack developer and the former owner of Hosting Data U.K. After graduating from the University of London with a Master’s Degree in IT, Alex worked as a developer, leading various projects for clients from all over the world for almost 10 years. He recently switched to being an independent IT consultant and started his technical copywriting career.

Submit an Article to CACM

CACM welcomes unsolicited submissions on topics of relevance and value to the computing community.

You Just Read

Goodhart’s Law Comes for Every Benchmark You Trust

© 2026 Copyright held by the owner/author(s).