🤖 AI Summary
Traditional vector-based associative memory models suffer from capacity degradation scaling as O(1/√n) due to superposition interference, limiting their ability to model long sequences. This work proposes Phase-Associative Memory (PAM), which represents states as complex-valued vectors in a complex Hilbert space, stores associations via outer products into matrix states, and retrieves them using conjugate inner products. PAM is the first method to natively integrate complex-valued hyperdimensional representations and conjugation operations into sequence modeling, fundamentally mitigating capacity decay. Under standard training protocols, PAM achieves a validation perplexity of 30.0 on WikiText-103 with approximately 100 million parameters—approaching the 27.1 attained by a same-scale Transformer—despite incurring roughly four times the arithmetic cost due to complex-number operations.
📝 Abstract
We present Phase-Associative Memory (PAM), a recurrent sequence model in which all representations are complex-valued, associations accumulate in a matrix state $S_{t}$ $\in$ $\mathbb{C}^{d \times d}$ via outer products, and retrieval operates through the conjugate inner product $K_t^* \cdot Q_t / \sqrt{d}$. At $\sim$100M parameters on WikiText-103, PAM reaches validation perplexity 30.0, within $\sim$10\% of a matched transformer (27.1) trained under identical conditions, despite $4\times$ arithmetic overhead from complex computation and no custom kernels. We trace the experimental path from vector-state models, where holographic binding fails due to the $O(1/\sqrt{n})$ capacity degradation of superposed associations, to the matrix state that resolves it. The competitiveness of an architecture whose native operations are complex-valued superposition and conjugate retrieval is consistent with recent empirical evidence that semantic interpretation in both humans and large language models exhibits non-classical contextuality, and we discuss what this implies for the choice of computational formalism in language modeling.