Training, Reading, and Editing Legible Transformers

2026-07-09

AI Summary

This work addresses the opacity of conventional Transformers by proposing a training framework that yields highly interpretable internal representations. By integrating readability regularization, a lower-bound variance loss at the channel level, unit-wise sparse activation, intra-layer rotational representation, and decorrelation constraints, the model learns to perform clear, sparse fuzzy-set operations within its units. Crucially, it replaces the handcrafted GELU partitioning with a learnable unit proportion mechanism that prevents representational degeneration. The resulting Transformer matches baseline performance while achieving unprecedented interpretability: 78% of feedforward layer operands and 50% of attention value channels function as explicit context detectors, and localized editing in deep layers exhibits a 50–184× improvement in locality compared to prior approaches.

Abstract

A transformer can be built from operators that are legible by construction -- bounded, named units that read as fuzzy set operations rather than dense activations -- but legibility must be pressed for during training, and the pressure has a failure mode. A crispness penalty meant to sharpen a bounded operator into a decisive detector instead collapses it into a dead constant. An identity, E[v(1-v)] = mu(1-mu) - var, shows why -- the penalty is a variance-minimizer blind to the difference between a live detector and a constant -- and names the fix: a per-channel variance floor, the target legibility metric written as a loss, which recovers both legibility and quality. A learned per-unit fraction then retires the hand-set reserved-GELU partition of prior work: given the choice the model keeps no unit as pure GELU and routes 87% of its load-bearing computation through crisp operators. The result is the most legible transformer we have built -- 78% of its feed-forward operands and 50% of its attention value channels are crisp-and-contextual detectors, and per-head legibility rises from 18% in shallow layers to 78% in deep ones. Read in the correct rotated per-layer frame, these units separate a clean detection (what a unit responds to) from a harder naming (what its output decodes to); and because the objective makes each unit crisp and sparse, edits to them are far more local -- 50-184x in the deep layers where the edit sites concentrate -- and can target explicit conjunctions a single neuron cannot express. Finally, a between-unit decorrelation pressure exposes a legibility dial: it trades a circuit's reuse for independence at no quality cost, turning concepts into single, surgically editable units and a prediction into a short explanation read off a handful of named operations. Quality holds at parity with a conventional baseline throughout.

Problem

• legibility
• transformer
• crispness penalty
• variance minimization
• interpretable units

Innovation

• legible transformers
• crispness penalty
• variance floor
• interpretable units
• model editing