Benchmark Contamination in plain English.

Benchmark contamination happens when the questions and answers from a standard AI test accidentally end up inside a model's training data, so the model "does well" on the benchmark not because it's genuinely capable, but because it has effectively memorized the answer key. Given that frontier models train on huge scrapes of the public internet, and popular benchmarks are themselves published on the public internet, some degree of contamination is a persistent risk across the industry. It's a serious problem because benchmark scores are the primary way labs, journalists, and buyers compare models — a contaminated score overstates real-world capability. Labs now routinely publish contamination-detection results alongside benchmark scores, and independent evaluators increasingly favor "held-out" or continuously refreshed benchmarks — new questions written after a model's training cutoff — specifically to sidestep the problem. Treat any single benchmark score with a healthy amount of skepticism.