Sparse Weight and Edge Circuit Discovery in Transformer-based Acoustic Models

2026-09-09

AI Summary

研究通过扩展DiscoGP框架至语音编码器，发现紧凑且高性能的计算子图，解释了现代语音基础模型的黑盒问题。

Abstract

Transformer-based foundation models are powerful but opaque, motivating Mechanistic Interpretation methods to uncover the black-box by identifying small computation subgraphs responsible for a task. DiscoGP is a joint weight-and-edge circuit discovery framework originally developed for text decoders. We extend DiscoGP to speech encoders and present, to our knowledge, the first circuit discovery study for modern speech foundation models. Across HuBERT and Wav2Vec 2.0 on several speech classification tasks, we find that the discovered circuits are extremely compact, yet often match or even exceed the performance of the full pretrained encoder with the same downstream head. Through ablations, we show that these circuits reflect pretrained computation rather than random structure or task-head artifacts. We also introduce a memory-efficient DiscoGP variant that reduces the GPU memory cost of edge-circuit discovery at runtime from quartic to cubic. Overall, our results broaden Mechanistic Interpretation beyond text decoders and show that circuit-level analysis can reveal both explanatory structure and unexpected functional behavior in speech encoders.

Problem

• Transformer-based Acoustic Models
• Mechanistic Interpretation
• Circuit Discovery

Innovation

• DiscoGP
• Speech Encoders
• Circuit Discovery
• Memory-Efficient