Evaluating LLMs (benchmarks, leaderboards, judges)

Why this matters

A senior LLM engineer is judged less by building a model than by measuring it. Wrong metric → wrong fine-tune → wasted GPU-months. The 2026 evaluation landscape is messy: benchmarks leak, judges are biased, Arena scores conflate skills. You need a worldview: which benchmarks tell you what, when LLM-as-judge is okay, and how to design a custom eval that actually measures what your product cares about.

Learning objectives

1. Distinguish capability, alignment, safety, and efficiency benchmarks.
2. Use lm-eval-harness and EleutherAI Eval Harness.
3. Understand the limitations of MMLU, MT-Bench, and Chatbot Arena.
4. Design a custom eval set with LLM-as-judge plus rule-based scorers.
5. Interpret a model card / technical report critically.

1. The evaluation taxonomy

Reporting just "MMLU = 75" is meaningless. Always report a cluster across categories.

2. Knowledge: MMLU and friends

MMLU (Hendrycks 2020)

57 multiple-choice subjects, 4 options each. Score = accuracy. Frontier models hit 85-90+. Was the headline metric for years.

Problems:

• Many leaked answers in pretraining corpora.
• 4-choice MCQ is gameable.
• Subject coverage is uneven.

MMLU-Pro (2024)

Harder, 10-choice, less leakage. Frontier models around 70-80.

GPQA (Rein et al., 2023)

Graduate-level science questions, "Google-proof" — even with web access, the average human takes a long time. Designed to be uncheatable. Frontier models 50-70+.

Reading the leaderboards

Hugging Face's Open LLM Leaderboard v2 aggregates MMLU-Pro, GPQA, MUSR, IFEval, BBH, MATH. Use it as a vibe check, not gospel.

3. Reasoning and math

• GSM8K (Cobbe 2021) — 8.5k grade-school math problems with chain-of-thought solutions. The standard maths benchmark.
• MATH (Hendrycks 2021) — 12.5k high-school competition problems; harder.
• AIME / IMO — olympiad-level. The 2024-2025 frontier (o1, R1) hit dramatic gains here.
• BBH (Suzgun 2022) — Big-Bench Hard subset; 23 reasoning tasks.
• MUSR — multi-step soft reasoning.

4. Code

• HumanEval (Chen 2021) — 164 small Python problems with unit tests. Saturated by 2024 (~95+%).
• MBPP — similar, 1k problems.
• LiveCodeBench — refreshed monthly to avoid contamination; running scores on recent contest problems.
• SWE-bench / SWE-bench Verified — real GitHub issues + test patches. The hardest mainstream coding benchmark.
• Aider's polyglot benchmark — multi-language code editing.

For self-study, run bigcode-evaluation-harness for HumanEval and SWE-bench-Lite.

5. Instruction following

IFEval (Zhou 2023)

Verifiable instruction tests: "respond in exactly 3 paragraphs," "include exactly 5 bullet points," "do not use the letter e." Rule-based scoring. Catches models that say yes but don't follow the instructions.

MT-Bench (Zheng 2023)

80 multi-turn questions; LLM judge (GPT-4) scores each from 1-10. Easy to run; biased by judge.

AlpacaEval 2 / LC AlpacaEval

Length-controlled win-rate vs GPT-4-Turbo as the reference. Less length-bias than MT-Bench.

6. Chatbot Arena

Massive crowdsourced preference battles between anonymous models. Produces an Elo-style ranking. Currently the most respected single signal for chat-model quality, because:

• Real humans, real prompts.
• No fixed eval set to memorise.
• Continuously updated.

Caveats:

• Heavy on conversational / opinion / creative tasks.
• Light on hard reasoning, code, agents.
• Style can dominate substance.

The right way to read it: "this model is preferred in conversational quality." Combine with capability benchmarks.

7. Long context

• RULER (Hsieh 2024) — multi-needle, multi-key, variable-difficulty NIAH variants.
• ZeroSCROLLS — long doc summarisation, QA.
• LongBench / LongBench-V2 — 21 tasks across long context.
• NIAH (gkamradt) — single needle, easiest test.

A model that says "1M context" should pass NIAH at 1M and RULER's harder variants. Many do not.

8. LLM-as-judge — when and how

LLM judges (gpt-4o, claude-sonnet, llama-3.1-70b) are practical for evaluating open-ended output. They have known biases:

• Position bias (slightly prefer A or B; randomise).
• Length bias (prefer longer; length-control).
• Self-preference (judges prefer their own family's outputs).
• Refusal preference (over-reward refusals).

Best practices:

1. Use the strongest judge you can afford.
2. Use 3-judge majority vote for borderline cases.
3. Position-randomise; report robustness.
4. Calibrate against human ratings on a small set.
5. Use rule-based judges where possible (regex, exact match, code-execution).

JUDGE_PROMPT = """\
You will be shown a question and two responses A and B.
Decide which response is more helpful, accurate, and follows instructions.

If both are equal, prefer the more concise one.
Output JSON: {"choice": "A"|"B"|"tie", "reason": "..."}.

Question: {question}
A: {a}
B: {b}
"""

Tools: arena-hard-auto, judges library (Lighteval), weights-and-biases LLM Eval, Promptfoo, OpenAI evals.

9. Custom evals — the most useful kind

Public benchmarks tell you about general capability. Your product demands a custom eval, with these properties:

1. Aligned with the user task — sample from real user queries.
2. Tracks both correctness and quality — combine rule-based and LLM-judged.
3. Includes adversarial / failure cases — long inputs, ambiguous queries, off-topic.
4. ~50-200 examples is enough; anything more is ROI-poor early.
5. Versioned in git, runnable in CI.
6. Cheap to run ($1-10 per evaluation cycle).

Pattern:

class EvalCase(TypedDict):
    name: str
    input: str
    expected: str | None
    rubric: str
    must_contain: list[str]
    must_not_contain: list[str]

def score_one(model, case):
    out = model(case["input"])
    rule_score = all(s in out for s in case["must_contain"]) and \
                 all(s not in out for s in case["must_not_contain"])
    judge_score = llm_judge(case["rubric"], case["input"], out)
    return rule_score, judge_score

Run on every model checkpoint; gate releases on regressions.

10. Reading a technical report

When a lab releases a model, read the report critically:

• Architectural changes — what is novel? What is "Llama-3 with our data"?
• Training data — provenance, dedup, decontamination claims.
• Compute — total FLOPs / GPU-days; reproducibility?
• Evals — which benchmarks reported, which omitted; few-shot vs zero-shot.
• Comparisons — fair baselines? same compute / data?
• Limitations / risks — usually in a small section near the end.

Key skill: spotting cherry-picked numbers. If a report shows GSM8K but not MATH, or AlpacaEval but not Arena, ask why.

Hands-on lab (4 hours)

eval_lab.ipynb:

1. Install lm-evaluation-harness. Run MMLU on Qwen2.5-3B-Instruct. Take ~30 minutes; verify accuracy.
2. Run IFEval on the same model. Inspect failed cases.
3. Implement an LLM-judge using gpt-4o-mini for 30 of your own prompts. Compute pairwise win-rate against Llama-3.2-3B-Instruct.
4. Build a rule-based judge that flags refusals, hallucinated tool calls, and off-topic responses. Combine with LLM-judge.
5. Score the same models with position-randomised judging. Compare results — note the difference if any.
6. Bonus: contribute a custom eval set (50 prompts) for a real product use case (your own).

Common pitfalls

1. Reporting one benchmark only — always cluster across categories.
2. Using saturated benchmarks (HumanEval at 95+%) — moves are noise.
3. Forgetting decontamination check — your fine-tune may have seen the test set.
4. Trusting a single LLM-judge run — high variance; do 3 runs.
5. Optimising for a public benchmark — the model overfits to that benchmark's style. Better: optimise for your custom eval.

Self-check

1. Why prefer GPQA over MMLU for frontier models?
2. Difference between AlpacaEval 2 and Chatbot Arena.
3. Three biases of LLM-as-judge?
4. What does IFEval test that MT-Bench doesn't?
5. What makes a good custom eval set?

References

• Hendrycks et al. (2020), "Measuring Massive Multitask Language Understanding" (MMLU).
• Wang et al. (2024), "MMLU-Pro."
• Rein et al. (2023), "GPQA: A Graduate-Level Google-Proof Q&A Benchmark."
• Cobbe et al. (2021), "Training Verifiers to Solve Math Word Problems" (GSM8K).
• Chen et al. (2021), "Evaluating Large Language Models Trained on Code" (HumanEval).
• Zheng et al. (2023), "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena."
• Hsieh et al. (2024), "RULER."
• HuggingFace Open LLM Leaderboard v2.
• LMSYS Chatbot Arena.
• Eleuther lm-evaluation-harness.