Local Causal Attribution of Chain-of-Thought Reasoning

2026-06-19

AI Summary

This work proposes AttriCoT, a novel method that enhances the transparency and safety of language models by enabling efficient causal attribution of Chain-of-Thought (CoT) units in a black-box setting within O(U) forward passes. By integrating structural causal models with perturbation analysis, AttriCoT performs localized causal evaluation of individual CoT units and quantifies their importance based on output probabilities. Experimental results across five datasets and four reasoning models demonstrate that AttriCoT produces attributions that more faithfully reflect the actual model behavior compared to existing approaches, effectively uncovering differences in reasoning structures across models and tasks.

Abstract

Understanding the causal structure of a language model's thought process is a problem of significant importance for both transparency and safety. In this work, we take a local approach toward this goal by analyzing the causal relationships among individual components, termed units, of a given, specific chain-of-thought trace. We construct a structural causal model on these units and relate each unit to the log probability of generating (subsequent) output units. Our algorithm, termed AttriCoT, is a black-box method that performs attribution by estimating importance parameters in the structural causal model using $O(U)$ forward passes through the model, where $U$ is the number of units. Evaluation of perturbation curves across 5 datasets and 4 reasoning models shows that AttriCoT produces attributions that are more faithful to the model's behavior than alternative methods. The attribution results also reveal notable differences in thought structure between models and domains.

Problem

• causal attribution
• chain-of-thought
• language models
• transparency
• safety

Innovation

• Chain-of-Thought
• Causal Attribution
• Structural Causal Model
• Black-box Method
• Model Interpretability