When Do Concepts Become Functionally Sufficient During Language-Model Training?

2026-08-15

AI Summary

This study addresses the unclear evolution of functional sufficiency regarding internal concepts in language models by proposing an interventional verification framework centered on this criterion. Through activation decomposition, sparse soft masking, and cross-checkpoint alignment, interpretability hypotheses are translated into functional tests involving reconstruction, decoding, and downstream tasks. Experiments across seven models demonstrate that downstream mask soft quality is significantly lower than reconstruction mask quality, accompanied by minimal predictive distribution shift. These findings elucidate the functional evolutionary dynamics of conceptual structures during training, providing empirical evidence and a novel paradigm for assessing the functional completeness of internal model representations.

Abstract

Understanding a model and its learning mechanisms in depth requires identifying when its internal structures become useful, rather than simply looking at the final state. We study this through concept dynamics: at each layer and checkpoint, we decompose activations, select sparse soft masks, and inject masked reconstructions into the model. Concept analysis is therefore tested functionally: a mask is useful only insofar as it preserves a target under intervention. We compare sufficiency for activation reconstruction, linear decodability, true downstream preservation, and checkpoint transfer under learned alignment. The framework treats decomposition assumptions as hypotheses rather than interpretability guarantees, monitoring functional sufficiency across checkpoints and source-to-final reconstructability under learned alignment. At the shared fixed-penalty operating point across seven models, downstream masks retain substantially less soft mass than reconstruction masks; predictive-distribution shifts remain small.

Problem

• concept dynamics
• functional sufficiency
• language model training
• internal representations

Innovation

• Concept Dynamics
• Functional Sufficiency
• Sparse Soft Masks
• Intervention-based Analysis
• Learned Alignment