Thought without systematicity? Evaluating reasoning models on rule induction tasks

2026-09-12

AI Summary

研究通过规则归纳任务评估推理模型的系统性，发现尽管模型能解决特定任务，但在结构等价变体上常失败，表明其缺乏系统性。

Abstract

A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept. Do reasoning models robustly exhibit such systematicity? If so, we would expect consistent performance on structurally equivalent variants of the same task. Here, we extend established rule induction tasks from cognitive science to assess the systematicity of thought in current reasoning models. Each task family has compositional structure that we use to create structurally equivalent task variations through task isomorphisms such as recombination and substitution. We find that despite being able to correctly solve a task, models often fail on structurally equivalent variants of the same task. These findings suggest that many model behaviors lack systematicity, rendering it difficult to robustly establish the cognitive abilities of reasoning models beyond the particular contexts they were evaluated in.

Problem

• systematicity
• reasoning models
• rule induction tasks
• structural equivalence

Innovation

• systematicity
• reasoning models
• rule induction tasks
• task isomorphisms