Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity

2026-06-16

AI Summary

This study investigates the impact of positional encoding on functional equivalence and linear mode connectivity in Transformer models. By leveraging theories of functional equivalence, group symmetry analysis, and alignment algorithms, the work systematically compares sinusoidal positional encoding with Rotary Position Embedding (RoPE) within the attention mechanism. It reveals for the first time that sinusoidal encoding preserves the original attention’s symmetry structure, whereas RoPE substantially weakens the symmetry group, thereby enhancing model expressivity. Furthermore, the choice of positional encoding directly determines the presence, absence, and variation patterns of linear mode connectivity. These findings elucidate the critical role of positional encoding design in shaping the geometric structure and optimization landscape of Transformer models.

Abstract

Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence. While this symmetry is well understood in classical fully connected and convolutional models, it becomes substantially more intricate in modern attention-based architectures. Existing analyses of multihead attention have largely focused on the vanilla formulation, overlooking positional encodings that fundamentally reshape architectural symmetries. In this work, we provide a formal study of functional equivalence in Transformers with positional encodings. Focusing on the two most widely used variants--sinusoidal and rotary positional encodings (RoPE)--we show that sinusoidal encodings preserve the equivalence structure of vanilla attention, whereas rotary encodings significantly reduce the symmetry group, thereby enhancing expressivity. This offers a principled explanation for the growing prominence of RoPE in practice. We further examine how positional encodings affect linear mode connectivity, and through an alignment algorithm, empirically demonstrate that the presence and variability of connectivity across Transformer settings crucially depend on the positional encoding.

Problem

• functional equivalence
• attention mechanism
• positional encodings
• linear mode connectivity
• Transformer

Innovation

• functional equivalence
• positional encodings
• rotary positional encoding (RoPE)
• linear mode connectivity
• Transformer symmetry