🤖 AI Summary
This work proposes PJ-RoPE, a novel learnable relative positional encoding framework that unifies Fourier phase, finite-order jets (including those with repeated roots), and unit-root affine recency within the algebraic structure of constant-coefficient difference modules. By decoupling feature transformation from bias kernels, introducing LC/rapidity coordinates, and incorporating an adaptive sector-diagnostic mechanism, the method stabilizes high-order jets and reveals task-specific preferences for positional structures. Experiments demonstrate that distinct tasks exhibit strong affinities for particular positional sectors; in small-scale language modeling, a clear boundary emerges between affine and recency-based encodings; in music sequence modeling, LC/affine variants achieve superior performance and implicitly encode higher-order corrections; and while LC coordinates enhance scale stability, they entail a trade-off in phase resolution.
📝 Abstract
We unify RoPE's Fourier phase, Jordan-RoPE's finite jets, and ALiBi's affine recency into a single learnable relative-position space, and study which regions of this space are selected by different tasks. PJ-RoPE is a Fourier-Jet-Affine formulation for relative attention, with an optional Poincare-type reading as the affine completion of a homogeneous Fourier-jet positional representation. Algebraically, the same primitives form a finite constant-coefficient difference module: simple roots of the lag-shift operator give Fourier/RoPE characters, repeated nonzero roots give Jordan/Fourier jets, and the repeated unit root gives ALiBi-like affine recency.
The framework separates scalar PJ-bias kernels from exact PJ-rotary feature transforms, introduces adaptive sector diagnostics, and uses LC/rapidity coordinates to stabilize high-order jets. Controlled probes verify sector containment and selection; small language runs expose an affine/recency boundary; music-token streams provide the clearest case where LC/affine variants remain strong while carrying measurable high-order corrections; and LC diagnostics show a scale-stability gain coupled to phase-resolution loss.