🤖 AI Summary
This work establishes fundamental limitations of non-negative kernel attention mechanisms, showing that their feature dimension must grow exponentially even to handle contexts as short as three tokens. Focusing on the exact modeling of Boolean inputs under the Min-IP task, the authors employ rank analysis, information-theoretic lower bounds, and constructive counterexamples within a framework of causal queries and position-dependent mappings. Their key contribution is the first proof that standard Softmax attention solves this task with only linear feature dimensionality, whereas non-negative kernel attention heads require at least $2^{\Omega(m)}$ dimensions. Furthermore, they derive an information transmission lower bound for multi-head, multi-layer models operating over finite alphabets.
📝 Abstract
Full attention exposes every token pair, whereas kernel attention compresses a sequence into a fixed-dimensional sketch. We show that this distinction becomes exponential at the first context length containing two competing candidates. On Min-IP over Boolean inputs, rank-one normalized kernel attention solves every sequence of length at most two exactly. In contrast, any single normalized nonnegative kernel-attention head that succeeds on all three-token sequences with error strictly below $1/2$ requires $2^{Ω(m)}$ features, even with arbitrary finite-dimensional tokenwise values and an arbitrary query-dependent affine readout. Dense softmax solves the same task with $m$-dimensional scores and constant temperature. The conclusion survives position-dependent token maps and a causal final query. As context length grows, the lower bound approaches the exact $2^m$-feature realization. Separately, for deterministic multihead, multilayer sketch models whose cross-token channels have finite alphabets, we prove a transcript lower bound linear in the number of independent answers and logarithmic in their alphabet size.