When AI models have seen the exam: benchmark contamination
The idea
Machine learning progress is mostly measured with benchmarksbenchmark A fixed set of tasks used to score and compare AI systems.: fixed sets of test questions that every new model is scored on. The method rests on one assumption, that the model has not seen the test questions during training. A student who has read the exam paper in advance is not showing what they have learned.
For modern large language modelslarge language model A computer program trained on very large amounts of text to predict and produce written language. that assumption is hard to guarantee. They are trained on enormous scrapes of the web, and popular benchmarks are published on the web, discussed in forums and copied into tutorials. When test items, or close paraphrases of them, end up in training data, the benchmark is said to be contaminated. A related problem is overfittingoverfitting When a model fits the quirks of the particular examples it was tuned on so closely that it does worse on new examples. to the benchmark: even without exact copies, years of choosing methods by their scores on one test set can tune a field to that set’s quirks.
A concrete example: GSM8k is a widely used set of grade school maths word problems. In 2024 Hugh Zhang and colleagues wrote a new set, GSM1k, built to match GSM8k on human solve rates, number of solution steps, answer size and other measures. If a model really can do grade school maths, it should score about the same on both.
Why it might change a decision
The prediction is simple and checkable. If a model’s benchmark score reflects contamination, it should fall on a fresh test set drawn the same way, and the fall should be larger for models more likely to have seen the original. That changes how a careful reader treats a leaderboard: a score on an old public benchmark says less than a score on a recently written or private one, and the gap between the two is itself informative.
How solid is this?
The evidence comes from a handful of studies using different methods, which point in a similar direction with important caveats.
Zhang and colleagues found that leading open and closed models dropped by up to 8% in accuracy on GSM1k compared with GSM8k, and that several model families showed signs of systematic overfitting. They also measured how likely each model was to generate GSM8k examples, and found a positive relationship with the size of its score gap (Spearman r² = 0.36), which suggests partial memorisation. The same abstract is careful about the other side: many models, especially frontier ones, showed minimal overfitting, and all models could solve new problems they had not seen. The paper appeared in the NeurIPS 2024 Datasets and Benchmarks track.
Shahriar Golchin and Mihai Surdeanu (2023) tackled detection without access to training data. Their method prompts a model with a dataset’s name, the partition (such as test) and the opening of an item, then checks whether the model completes it exactly or nearly so. Against judgements by human experts, their best version was 92% to 100% accurate across seven datasets. They reported that GPT-4 appeared to be contaminated with three of them: AG News, WNLI and XSum. A limitation is that the method only catches items a model can reproduce; a model could have learned from leaked items without reciting them.
The older analogue complicates the story. In 2019 Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt and Vaishaal Shankar rebuilt test sets for two image classification benchmarks, CIFAR-10 and ImageNet, by closely following the original collection procedures. Across a broad range of models, accuracy fell by 3% to 15% on CIFAR-10 and 11% to 14% on ImageNet. That looks like overfitting, but the authors argued otherwise: models that did better on the old test sets also did better on the new ones, with gains that were even larger on the new sets. They attributed the drops to models struggling with images slightly harder than those in the originals, not to a decade of test set reuse.
That points to the strongest competing explanation for any drop on a fresh test set: the new set may simply be a little different or harder, however carefully it is matched. Zhang and colleagues tried to rule this out by matching on human solve rates, but no matching is perfect. The popular claim that leaderboard scores are mostly memorisation outruns the data. The narrower claim, that contamination inflates some scores for some models by measurable amounts, is supported, with the size varying a lot between models.
Sources
- Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., Lyu, Q., Hendryx, S., Kaplan, R., Lunati, M., & Yue, S. (2024). A careful examination of large language model performance on grade school arithmetic. NeurIPS 2024 Datasets and Benchmarks Track. arXiv:2405.00332
- Golchin, S., & Surdeanu, M. (2023). Time travel in LLMs: Tracing data contaminationdata contamination When the questions used to test an AI model, or close copies of them, were already in the material it learned from. in large language models. arXiv:2308.08493
- Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019). Do ImageNet classifiersclassifier A model that sorts inputs into categories, such as flagged or not flagged. generalize to ImageNet? Proceedings of the 36th International Conference on Machine Learning, PMLR 97, 5389-5400. proceedings.mlr.press/v97/recht19a