Shared circuits predict whether LLMs generalize across formats in arithmetic reasoning

2026-09-03

AI Summary

研究通过归因修补方法定位模型解决数字和文字算术问题的电路，并验证了电路重叠程度可以预测模型在不同格式间的泛化能力。

Abstract

In many forms of reasoning, including arithmetic reasoning, generalizing across superficial changes in input format is effortless for humans: anyone who can solve 2+5 can also solve 'two plus five'. In contrast, LLMs are more brittle to surface variations of the prompts: for example, they solve numeric arithmetic problems almost perfectly but are substantially less accurate on verbal renditions of the same problems. Here, we ask whether generalization across formats can be predicted from the models' internals. Using attribution patching, we first independently localize the circuit that each model recruits to solve numeric arithmetic problems (2+5) vs. verbal ones, in three languages: English ('two plus five'), Spanish ('dos más cinco'), and Italian ('due più cinque'); then, we test whether overlap with the model's own numeric circuit predicts its generalization to the verbal formats. Indeed, we find support for this idea at three levels: circuit overlap accounts for the relative difficulty of the three verbal formats, for which models generalize best, and for which items are solved correctly, rivaling supervised probes while requiring no labeled data.

Problem

• LLMs
• arithmetic reasoning
• generalization
• input format
• surface variations

Innovation

• attribution patching
• circuit overlap
• generalization across formats