🤖 AI Summary
This work investigates the necessity of distinct query (Q), key (K), and value (V) projections in Transformer models and proposes three parameter-sharing strategies: Q–K=V, Q=K–V, and Q=K=V. To mitigate symmetric attention patterns arising from shared projections, the authors introduce a two-dimensional positional encoding scheme. Empirical results demonstrate that the Q–K=V configuration incurs only a 3.1% increase in perplexity on language modeling tasks while reducing KV cache memory by 50%. When combined with multi-query or grouped-query attention (MQA/GQA), this approach achieves up to 96.9% KV cache compression, maintaining competitive or superior performance across both vision and language benchmarks. The method substantially lowers memory overhead, seamlessly integrates with existing head-sharing mechanisms, and offers a promising pathway for efficient on-device deployment.
📝 Abstract
Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role. However, the individual contribution of these three projections and the impact of omitting some remain poorly understood. We systematically evaluate three projection sharing constraints: a) Q-K=V (shared key-value), b) Q=K-V (shared query-key), and c) Q=K=V (single projection). The last two variants produce symmetric attention maps; to address this, we also explore asymmetric attention via 2D positional encodings. Through experiments spanning synthetic tasks, vision (MNIST, CIFAR, TinyImageNet, anomaly), and language modeling (300M and 1.2B parameter models on 10B tokens), we discovered that our transformers perform on par or occasionally better than the QKV transformer. In language modeling, Q-K=V projection sharing achieves 50% KV cache reduction with only 3.1% perplexity degradation. Crucially, projection sharing is complementary to head sharing (GQA/MQA): combining Q-K=V with GQA-4 yields 87.5% cache reduction, while Q-K=V + MQA achieves 96.9%, enabling practical on-device inference. We show that Q-K=V preserves quality because keys and values can occupy similar representational spaces and attention operates in a low-rank regime, whereas Q=K-V breaks attention directionality. Our results systematically characterize projection sharing as an underexplored instance of weight tying in attention, with direct, quantifiable inference memory benefits, particularly valuable for edge deployment. The code is publicly available at https://github.com/anushamadan02/Do-Transformers-Need-3-Projections