Scalable Circuit Learning for Interpreting Large Language Models

2026-06-15

AI Summary

Existing intervention-based circuit learning methods incur substantial computational costs when applied to high-dimensional, sparse features from sparse autoencoders (SAEs), hindering efficient discovery of interpretable semantic mechanisms within large language models. This work proposes CircuitLasso, which introduces sparse linear regression into circuit learning for the first time to model structured dependencies among SAE features and efficiently trace the propagation pathways of semantic information. CircuitLasso achieves circuit reconstruction accuracy comparable to state-of-the-art intervention methods while significantly reducing computational overhead. Furthermore, it demonstrates strong performance in domain generalization tasks, matching or exceeding existing approaches at a fraction of the computational cost.

Abstract

A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior. However, raw neurons are polysemantic, making learned circuits hard to interpret. Sparse autoencoder (SAE) features alleviate this, but their high dimensionality makes existing intervention-based circuit learning methods computationally prohibitive. We propose CircuitLasso, a scalable circuit-learning approach based on sparse linear regression. CircuitLasso recovers circuits whose structural accuracy matches that of state-of-the-art intervention-based methods on the benchmark data, at a fraction of the computational cost. For interpretability, CircuitLasso efficiently uncovers relationships among SAE features, showing how human-interpretable semantic features propagate through the model and influence its predictions. Finally, we validate the utility of our learned circuits by leveraging their insights to achieve comparable performance at substantially lower cost on a domain-generalization task.

Problem

• mechanistic interpretability
• sparse circuits
• large language models
• sparse autoencoder features
• scalable circuit learning

Innovation

• CircuitLasso
• sparse linear regression
• mechanistic interpretability
• sparse autoencoder features
• scalable circuit learning