Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental science

2026-08-19

AI Summary

研究通过引入AtmosCoder-Bench评估环境科学中语言模型的计算过程，揭示了多选题高估准确率及模型在多步骤计算和特定条件下的应用问题。

Abstract

Large language models are increasingly used for quantitative work in the environmental sciences, yet existing evaluations score only final answers, leaving calculation process unobserved. Here we introduce AtmosCoder-Bench, an execution-grounded benchmark that makes the calculation process visible. Built through a transferable semi-automated pipeline (436 problems, 3,910 variants, 7,029 graded quantities), every problem is validated to be unambiguous and human-solvable, with uniquely verifiable answers. We find that (i) multiple-choice formats inflate measured accuracy by at least 12 percentage points; (ii) many failures arise not from missing knowledge but from models failing to apply known formulas and constraints consistently throughout multi-step computation; and (iii) even frontier models remain weak when task-specific conditions invalidate familiar methods, often reverting to canonical solution patterns rather than adapting methods to the relevant physical regime, leaving expert oversight essential.

Problem

• language models
• environmental science
• calculation process
• hidden failures
• execution-grounded evaluation

Innovation

• execution-grounded evaluation
• calculation process visibility
• AtmosCoder-Bench