15  Benchmark Catalog

Living document. This catalog covers benchmarks prominent at the time of last review. The evaluation landscape moves fast — benchmarks are released, saturated, and superseded on 12–24 month cycles. Entries will be updated each quarterly review; treat any specific result or saturation claim as time-stamped, not permanent.

For the conceptual framework — why evaluation has passed through distinct generations and what each generation revealed — see Evaluation & Benchmarks.

15.1 Generation 1 — Static Accuracy (2018–2022)

Fixed-distribution benchmarks that defined the first era of LLM evaluation. All saturated or broken by contamination within two years of release.

Benchmark	Year	Measures	How it broke
GLUE / SuperGLUE	2018/19	NLU tasks (NLI, QA, coreference)	Models saturated within months of release
MMLU	2020	57-subject multiple-choice knowledge	Contamination; frontier models now exceed human average
HumanEval	2021	Python code generation (pass@k)	pass@k gameable; real-world correctness diverges from pass rate
BIG-bench	2022	204 diverse tasks, difficulty-scaled	Heterogeneous scoring; no safety dimension; marks end of static era

15.2 Generation 2 — Holistic Evaluation (2022–2023)

Benchmark	Year	Measures	Notes
HELM (Liang et al., 2023)	2022	Accuracy, calibration, robustness, fairness, toxicity, efficiency across shared scenarios	First multi-dimensional framework; exposed accuracy–robustness decorrelation

15.3 Generation 3 — Adversarial (2023–2024)

Benchmark	Year	Measures	Notes
HarmBench (Mazeika et al., 2024)	2023	Attack success rate across 18 attack methods × 5 harm categories × 33 target LLMs	Standardized attack + defense evaluation in a single framework
Agent Security Bench (Zhang et al., 2025)	2024	Direct + indirect injection across 10 scenarios, 10 attack methods, 400+ tools	No evaluated defense eliminated the attack surface

15.4 Generation 4 — Agentic (2024–present)

Benchmark	Year	Measures	Notes
AgentDojo (Debenedetti et al., 2024)	2024	Attack success rate + utility preservation on realistic tool-using agent tasks (email, calendar, banking API)	Frontier models complete 60–70 % of tasks; vulnerable to injection in 30–60 % of attack scenarios; input filtering cuts both attack and task success
Agent Security Bench (ASB) (Zhang et al., 2025)	2024	10 scenarios × 10 attack methods × 400+ tools; unified attack success rate	Direct injection 84 % ASR on undefended agents; indirect injection 60–80 %; no evaluated defense eliminated attack surface
τ-bench / service-task reliability	2024	Agent reliability across stochastic, multi-turn service tasks (airline, retail)	Measures over-action failures — when agent should pause, refuse, escalate but does not

15.5 Generation 5 — Dangerous Capabilities (2024–present)

A positive result is a warning signal, not a capability gain.

Capability	Evaluation focus	Representative approach
CBRN	Uplift beyond freely available information on chemical, biological, radiological, nuclear weapons	Structured elicitation against domain-expert counterfactuals
Cyber	Novel exploit generation, autonomous attack chain completion	CTF benchmarks; autonomous pen-test environments
Autonomy / AI R&D	Long-horizon task completion, research engineering at human-expert level	METR RE-Bench (Wijk et al., 2024)
Self-proliferation	Copy and deploy own weights, acquire compute autonomously	Controlled sandbox with monitored exfiltration opportunity
Deception	Behavior changes between evaluation and deployment context	Consistency probing; activation analysis (Panfilov et al., 2025)

Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). AgentDojo: A dynamic environment to evaluate attacks and defenses for LLM agents. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks.

Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research.

Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., & Hendrycks, D. (2024). HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. International Conference on Machine Learning (ICML).

Panfilov, A., Kortukov, E., Nikolić, K., Bethge, M., Lapuschkin, S., Samek, W., Prabhu, A., Andriushchenko, M., & Geiping, J. (2025). Strategic dishonesty can undermine AI safety evaluations of frontier LLMs. arXiv Preprint arXiv:2509.18058.

Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., Ericheva, E., Garcia, K., Goodrich, B., Jurkovic, N., Karnofsky, H., Kinniment, M., Lajko, A., Nix, S., Sato, L., … Barnes, E. (2024). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv Preprint arXiv:2411.15114.

Zhang, H., Huang, J., Mei, K., Yao, Y., Wang, Z., Zhan, C., Wang, H., & Zhang, Y. (2025). Agent security bench (ASB): Formalizing and benchmarking attacks and defenses in LLM-based agents. International Conference on Learning Representations (ICLR).