ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions

2026-08-31

AI Summary

为解决前沿大模型科学推理能力评估难题，通过构建ScienceArena基准，采用专家审核的数字化流程和自动评分方法，评估多个模型在最新科学竞赛题上的表现。

Abstract

Benchmark saturation and data contamination increasingly obscure genuine scientific reasoning in frontier LLMs. We introduce \textsc{ScienceArena}, an olympiad-style benchmark from thirteen public science competitions in physics, chemistry, and biology, including IPhO and IChO 2025--2026, IBO 2023, USAPhO 2026, and USNCO 2025. Its open-ended, multi-step problems use process-credit rubrics, making faithful scoring difficult. We build ScienceArena through an expert-audited digitization pipeline that converts official exams, figures, solutions, and rubrics into structured items verified by olympiad medalists. To scale evaluation beyond costly human grading, we calibrate LLM-as-judge against medalist ground truth on archived answers from five models across IPhO and IChO; two strong judges stay within one point of expert total scores. Medalist notes show that failures often stem from visual grounding, structure fidelity, and global problem control rather than missing terminology. Evaluating fourteen recent LLMs with interleaved solving, we find that top models obtain medal-equivalent rubric scores on several public international exams, while chemistry and long-horizon consistency remain key bottlenecks. We provide an interactive \href{https://science-arena.onrender.com/}{demo}.

Problem

• Benchmark Saturation
• Data Contamination
• Scientific Reasoning
• LLMs
• Olympiad Competitions

Innovation

• ScienceArena
• olympiad-style benchmark
• LLM-as-judge
• process-credit rubrics
• expert-audited digitization