Scaling Inherently Interpretable Language Models

2026-08-06

AI Summary

This work challenges the conventional trade-off between interpretability and performance in language models by introducing a novel paradigm that treats interpretability as a training constraint jointly optimized with the language modeling objective. The proposed approach enhances both capability and interpretability simultaneously within autoregressive and diffusion architectures. Built upon Steerling-8B—a diffusion-based language model equipped with causal attention masking—it integrates input attribution, concept attribution, and data provenance techniques to achieve disentangled representations aligned with human-understandable concepts, enabling closed-loop interventions. Experimental results demonstrate that Steerling-8B matches or exceeds the performance of leading open-source models while using only one-half to one-sixteenth of their computational budget, thereby validating that interpretability can scale synergistically with model size rather than being sacrificed for it.

Abstract

Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish. In this work, we challenge this premise. Rather than reverse-engineering a model, we make interpretability a constraint of the training pipeline, optimized alongside the language modeling objective. Across three orders of magnitude of compute, on both autoregressive and diffusion language models, interpretability scales with capability rather than against it. Surprisingly, model representations become more disentangled and aligned with human-understandable concepts with scale. We instantiate the training-time recipe with Steerling-8B, a diffusion language model with a causal attention mask. For any group of generated tokens, Steerling-8B attributes the output to relevant input tokens, human-understandable concepts, and training data. This enables closed-loop intervention: diagnose an output through its concept or feature attribution, retrieve similar training data, and correct the behavior through concept steering without retraining. Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale.

Problem

• interpretability
• language models
• scaling
• training-time interpretability
• disentangled representations

Innovation

• inherent interpretability
• training-time interpretability
• concept steering
• disentangled representations
• diffusion language models