Meta’s own technical report for Llama 2 disclosed something most benchmark leaderboards never mention: a meaningful share of the questions in MMLU, one of the most widely quoted AI benchmarks, overlapped with the model’s own training data. The model wasn’t being tested on unseen material. In places, it was being asked questions it had already encountered, sometimes with more than 80 percent of the exact wording matching.
This isn’t a minor asterisk. It’s a structural problem that touches nearly every widely cited AI benchmark, and it means a leaderboard score is measuring something more complicated than the plain “how smart is this model” story usually told about it.
What Do AI Benchmarks Actually Measure?
AI benchmarks like MMLU, HumanEval, and GSM8K measure how well a model performs on a fixed, published set of test questions, under the assumption that the model has never seen those exact questions before. That assumption breaks down when benchmark questions leak into a model’s training data, a problem called contamination, which can inflate scores without reflecting any real improvement in the model’s underlying ability.
MMLU, for instance, covers 57 subjects with multiple-choice questions meant to test broad knowledge. HumanEval tests whether a model can write working code for a set of programming problems. Both are widely quoted the moment a new model launches. Neither number tells you, on its own, whether the model actually reasoned its way to the answer or simply recalled having seen it during training.
The Contamination Problem, With Real Numbers
Data contamination happens when benchmark questions and their answers end up inside a model’s training data, usually because the benchmark has existed publicly online long enough for web crawlers to pick it up. This isn’t necessarily deliberate. Models are trained on huge swaths of scraped internet text, and popular benchmarks get discussed, copied, and reposted across the web for years after release.
The scale of this is documented, not speculative. One contamination audit found roughly 29 percent of MMLU questions showed contamination signals when cross-checked against training data. Meta’s own Llama 2 technical report separately disclosed that more than 16 percent of MMLU items overlapped with its training set, with a portion showing severe leakage. When a benchmark question has effectively already been seen, a high score on it proves memorization at least as much as it proves reasoning, and there’s often no clean way to separate the two from the score alone.
MMLU Also Has Known Answer-Key Errors
Contamination isn’t the only issue. A separate, independent problem is that MMLU contains actual errors in its answer key: questions where the labeled “correct” answer is wrong, ambiguous, or debatable. This is well enough documented that it led directly to the creation of corrected replacement benchmarks, including MMLU-Pro and MMLU-Redux, built specifically to fix flawed items in the original.
A similar pattern shows up in HumanEval, the standard coding benchmark, where researchers found the test cases used to grade submissions were often incomplete, missing edge cases a genuinely correct solution needs to handle, which led to the creation of an expanded version called HumanEvalPlus.
Why Scores Cluster So Closely Together
Once contamination and labeling errors are accounted for, another pattern becomes harder to ignore: top models on MMLU today often differ by only a couple of percentage points. That gap is small enough to fall inside normal measurement noise, meaning the ranking order between two closely scored models on a leaderboard may not reflect any reliable difference in capability at all.
What This Means When You’re Comparing Models
| What a benchmark score can tell you | What it can’t reliably tell you |
|---|---|
| Rough relative performance on the specific test format used | Real-world performance on your actual task |
| Whether a model improved dramatically or barely at all | Whether a small, close-score difference is meaningful or noise |
| Performance on the benchmark’s specific subject coverage | Performance on subjects or formats the benchmark doesn’t cover |
| A general sense of a model generation’s capability | Whether the score reflects genuine reasoning or memorized test content |
None of this means benchmark scores are worthless. It means treating a leaderboard number as a precise, final ranking is a mistake the numbers themselves don’t support, especially once contamination rates and labeling errors are taken into account.
Questions Worth Asking Instead of Trusting a Leaderboard
- Has the benchmark been checked for contamination, or is it an older, widely circulated test that’s had years to leak into training data?
- Does the task you actually care about resemble the benchmark’s format, or are you extrapolating from a multiple-choice knowledge test to a completely different kind of real-world task?
- How close are the scores you’re comparing? A one- or two-point gap on a benchmark with known measurement noise is a weak basis for choosing between two models.
- Is there a newer, decontaminated version of the same benchmark (like MMLU-Pro or MMLU-Redux) with more current results available?
Frequently Asked Questions
Does a high MMLU score mean a model is smarter? It means the model performed well on that specific test, which is a weaker claim than it sounds like. Given documented contamination and labeling errors in MMLU, a high score reflects some mix of real capability and test familiarity that the score alone can’t separate.
Why do labs still report contaminated benchmarks if the problem is known? Partly because comparable historical data exists for these benchmarks going back years, and partly because decontaminated replacements are newer and not yet universally adopted. Reporting a legacy benchmark alongside a newer, cleaner one is increasingly common practice.
Is HumanEval a reliable way to judge coding ability? It’s a reasonable rough signal, but its original test cases were found to be incomplete in ways that let flawed solutions pass, which is why HumanEvalPlus, with more thorough test cases, was created.
Should I ignore benchmark scores entirely? No. They’re still useful for spotting large, obvious differences between models or generations. The mistake is treating small differences as meaningful, or treating any single score as proof of real-world quality on your specific task.
What’s a better way to evaluate a model for my own use case? Testing it directly on examples that resemble your actual task, which is a more reliable signal than any general-purpose benchmark, since no published benchmark can account for the specifics of what you’re actually trying to do.
The Practical Takeaway
A benchmark score is evidence, not proof, and the size of the gap between “evidence” and “proof” here is larger than most leaderboard coverage admits. Contamination rates in the double digits and documented labeling errors mean a small difference in reported scores often isn’t a meaningful difference in capability. This is a large part of why hands-on testing against real tasks, not benchmark scores, sits at the center of our own review methodology: a number on a leaderboard can’t tell you how a model handles your specific document, your specific codebase, or your specific question the way actually running it can. For a related mechanism that benchmarks systematically struggle to capture, see our explainer on why AI models hallucinate, and for why a model’s context handling has the same advertised-versus-actual gap as a benchmark score, see our guide to context windows.