LLM · EVALUATION
How LLM Benchmarks Work (and Why Scores Mislead)
An LLM benchmark is a fixed set of test items plus a rule for scoring the answers, run the same way on every model so the results can be compared as one number. The number is only as meaningful as the test behind it: if the test items leaked into the training data, the score partly measures memory.
A fixed test and a scoring rule
A benchmark has three parts: test items, a way of presenting them, and a rule for scoring the answers. Every model gets the same items in the same format and is marked by the same rule. That sameness is the whole point. It turns "this model seems good" into a number two models can be compared on.
A run goes in four steps. The benchmark supplies the items, for example questions with a known correct answer. Each item is wrapped in a prompt and sent to the model. The rule checks the model's answer. The per-item results are then combined, usually into a share of items answered correctly.
MMLU, introduced by Hendrycks and colleagues in 2020, is a concrete case. It has 15,908 questions across 57 subjects, from elementary mathematics to US history and law, each with four answer choices. The prompt opens with "The following are multiple choice questions (with answers)" and the subject. The model's probabilities for the letters A, B, C and D are compared, and the highest one counts as its answer. The score is the share of questions it gets right.
Three ways a benchmark scores an answer
Multiple choice, as in MMLU, is the cheapest: the answer is one letter, so scoring is an exact match with the key. It measures whether the model picks the right option. It does not measure whether the model can write a good explanation.
Code tasks are scored by running the answer. HumanEval, from Chen and colleagues in 2021, has 164 handwritten programming problems. Each gives a function signature and a description, and the model writes the function. The code is executed against unit tests, an average of 7.7 per problem, inside a sandbox. Its score is called pass@k: the model produces k attempts per problem, and the problem counts as solved if any one attempt passes the tests. pass@1 means one try, pass@10 allows ten.
Arenas replace the answer key with people. In Chatbot Arena, described by Chiang and colleagues in 2024, a visitor writes any prompt and sees two anonymous answers side by side. They vote for one, or for a tie, or for "both are bad". Model names appear only after the vote. Many votes are turned into a ranking with a statistical model; the paper uses Bradley-Terry. There is no key, so the score reflects what voters preferred, and the authors expect those voters to be mostly LLM hobbyists and researchers.
When the test was in the training data
An LLM is trained on enormous amounts of text collected from public sources. Many benchmarks are public too. When test items end up in the training data, the problem is called benchmark leakage or contamination. Zhou and colleagues, in 2023, describe it as data related to evaluation sets being used for training. It can happen without anyone intending it, because pre-training data is often prepared before the model is tested.
A benchmark assumes the model has not seen the items. If it has, a correct answer may be recall, not skill. In the experiments of Zhou and colleagues, leakage could dramatically boost evaluation results, which makes comparisons between models misleading.
Test authors know this and work around it. The HumanEval authors wrote their problems by hand because the models are trained on a large fraction of GitHub, where solutions to many public problems already exist. The GPT-3 paper (Brown and colleagues, 2020) removed overlap with benchmarks from its training data, then found a bug in its filter that let some through, and retraining was too costly. It reported that contamination had minimal effect on most datasets and flagged the few that might be inflated. Leakage does not always inflate a score, but the reader usually cannot tell from the number alone.
Other ways a score misleads
Saturation. When strong models all score near the top, the test stops separating them. The authors of MMLU-Pro wrote in 2024 that performance on MMLU had begun to plateau, and built a harder version with ten answer options instead of four.
Prompt format. Sclar and colleagues changed only the formatting of few-shot prompts, such as separators and casing, without changing the meaning. Accuracy for one open model varied by up to 76 points, and the effect persisted with larger models, more examples and instruction tuning. A score belongs to a model plus a prompt format, not to the model alone.
Errors in the key. A team re-annotated 5,700 MMLU questions and estimated that 6.49% of the benchmark's questions contain errors.
Narrow coverage. A test measures what it contains. The HELM authors found that, before their 2022 work, models had been evaluated on 17.9% of their core scenarios on average, so they scored 30 models on the same scenarios with seven metrics, including calibration, robustness and bias, not accuracy alone.
Optimizing for the test. When a score becomes the target, effort goes into raising the score instead of the quality it was meant to indicate. This is often called Goodhart's law. Leakage is one route to it.
How to read a leaderboard
A leaderboard is a table of benchmark scores sorted from high to low. It answers a narrow question: which model scored highest on this test, in this setup, on this date. "Which AI model is best?" has no answer until you name a task, such as writing code, summarizing contracts or translating Spanish.
Start with the task, then find the test that resembles it. Check what the items look like and how they are scored. Check whether the test set is public, which makes leakage possible. Check whether top scores sit so close together that the order means little. Check whether the prompt format and settings are stated. Treat a vote-based ranking and an answer-key score as different measurements of different things.
The most useful benchmark is often your own: a small set of real examples from your task, run through each model and scored the same way each time. This page names no model scores on purpose. They change with every release, and a number without its test is exactly what misleads.
FAQ
- What is a benchmark in AI?
- A benchmark is a fixed set of test items plus a scoring rule, run the same way on every model. The output is a number, such as the share of questions answered correctly, that lets two models be compared on the same test.
- What does MMLU measure?
- MMLU measures accuracy on 15,908 multiple-choice questions across 57 subjects, from elementary mathematics to law. The model picks one of four answers per question, and the score is the share it gets right. It is a test of picking the right option on exam-style questions, not of writing or long tasks.
- Which AI model is best?
- No model is best without a task. A leaderboard only says which model scored highest on one test, in one setup, at one date. Pick the test closest to your task, then run a few of your own real examples through the candidates.
- What is Chatbot Arena, and is it the same as LMArena?
- Chatbot Arena is a platform, described in a 2024 paper, where people vote between two anonymous model answers to a prompt of their own choosing. When last checked on 2026-10-11, the lmarena.ai address redirected to arena.ai, so the same project now appears under different names. The method is what matters: the ranking comes from human votes, not an answer key.
- Can a model cheat on a benchmark?
- Test items can end up in a model's training data, often by accident, which is called contamination or leakage. Research from 2023 found that leakage can dramatically boost evaluation results. A score from a test the model has seen partly measures memory instead of skill.
Sources
- Hendrycks et al., 2020, "Measuring Massive Multitask Language Understanding" (MMLU)
- Chen et al., 2021, "Evaluating Large Language Models Trained on Code" (HumanEval, pass@k)
- Chiang et al., 2024, "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference"
- Zhou et al., 2023, "Don't Make Your LLM an Evaluation Benchmark Cheater"
- Brown et al., 2020, "Language Models are Few-Shot Learners" (GPT-3, contamination analysis)
- Sclar et al., 2023, "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design"
- Wang et al., 2024, "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark"
- Gema et al., 2024, "Are We Done with MMLU?" (MMLU-Redux)
- Liang et al., 2022, "Holistic Evaluation of Language Models" (HELM)
Related
Last updated 2026-10-11