CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction

2026-04-28

AI Summary

Existing code reasoning benchmarks focus solely on the correctness of final outputs from a single implementation, failing to assess models’ ability to consistently reason about functionally equivalent implementations or intermediate execution states. This work proposes CoRE, the first fine-grained evaluation framework centered on implementation invariance and process transparency. CoRE constructs test cases using multiple functionally equivalent code variants and explicit intermediate state tracking to systematically evaluate the robustness and faithfulness of mainstream large language models in code reasoning. Experimental results reveal that current models exhibit significant performance fluctuations across equivalent implementations and frequently produce correct final outputs without accurately reasoning through intermediate states, exposing critical limitations in their surface-level execution behavior and lack of robust reasoning capabilities.

Abstract

Despite strong performance on code generation tasks, it remains unclear whether large language models (LLMs) genuinely reason about code execution. Existing code reasoning benchmarks primarily evaluate final output correctness under a single canonical implementation, leaving two critical aspects underexplored: (1) whether LLMs can maintain consistency to functionally equivalent implementations, and (2) whether LLMs can accurately reason about intermediate execution states. We introduce \textbf{CoRE}, a \textbf{Co}de \textbf{Re}asoning benchmark that evaluates code reasoning through \textbf{implementation invariance} and \textbf{process transparency}. Extensive evaluations on eight frontier LLMs reveal two fundamental limitations. First, models exhibit a substantial \textbf{robustness gap}, with performance varying significantly across equivalent implementations. Second, we observe \textbf{superficial execution}, where models arrive at correct final outputs without correctly reasoning about intermediate execution states. Together, these findings demonstrate that output-only evaluations are insufficient for assessing code reasoning and position CoRE as a necessary benchmark for evaluating robust and faithful code reasoning.\footnote{Data and code are available at https://github.com/ZJUSig/CoRE.}

Problem

• code reasoning
• implementation invariance
• process transparency
• large language models
• execution states

Innovation

• code reasoning
• implementation invariance
• process transparency
• robustness gap
• intermediate execution states