Legible-by-Construction: Attention and End-to-End Transformers

2026-07-05

AI Summary

This work addresses the limited interpretability of traditional Transformer attention mechanisms, which obscure the model’s decision logic. The authors propose a novel reformulation of the Value vectors in the attention module by incorporating Sigmoid or Boolean activation functions to impose sparsity and encourage extreme activation patterns. This transforms Value vectors into context-sensitive, semantically precise feature detectors while preserving output projection flexibility for selective responses. Integrated with existing interpretable feedforward layers, this approach yields the first end-to-end interpretable Transformer architecture. In a 125M-parameter model, 44%–62% of Value channels exhibit clear, selective detection behavior without compromising language modeling performance relative to baseline models, thereby achieving full-model interpretability without sacrificing accuracy.

Abstract

A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality. That left the other half of the transformer opaque. Here we carry the same idea into attention and join the two into one model. The mechanism is minimal: a head's value is passed through a sigmoid, so each value channel becomes a readable detector of whether a feature holds at a token. This adds no parameters and leaves the standard head otherwise untouched. A Boolean variant goes further, restructuring the value into an explicit within-token intersection and negation-capable set-difference. In both designs the output projection is left free, not tied to the vocabulary, which is the load-bearing decision: bounding what a head detects while leaving what it writes unconstrained yields selective detectors, whereas constraining the write does not. A bounded value is shaped into a readable detector by two selectivity pressures - one for sparse firing, one for decisive firing at the rails - and which a design wants is not universal. Across five specialized-attention designs at 125M parameters, 44 to 62 percent of value channels become crisp, contextually selective detectors, and their legibility rises with depth rather than crystallizing only on punctuation. Language-model quality is at parity with a conventional baseline. Finally, we couple the Boolean attention to the legible feed-forward layer and train an end-to-end legible-by-construction language model at benchmark parity: its feed-forward units are named set and quantifier operations throughout, and we can take a token it generates and read the named units that compose to produce it.

Problem

• legibility
• transformer
• attention mechanism
• interpretable AI
• language model

Innovation

• legible-by-construction
• interpretable attention
• Boolean transformers
• fuzzy set operations
• selective detectors