Controlled Dynamics Attractor Transformer

2026-06-13

AI Summary

This work addresses the lack of biologically plausible continuous-time inference dynamics in existing Transformers and the difficulty of integrating classical continuous attractor neural networks (CANNs) with modern attention mechanisms. To bridge this gap, the authors propose a novel energy-based attention framework that, for the first time, incorporates CANN-inspired excitation-inhibition modulation into the Transformer architecture. By combining von Mises–Fisher attention with Hopfield-like refinement energy, the model establishes dissipative attractor dynamics governed by topological constraints. This formulation enables structured and controllable continuous-time reasoning, achieving state-of-the-art performance across multiple benchmark tasks in graph anomaly detection and graph classification.

Abstract

Transformer architectures have dramatically advanced representation learning and inference in deep models through self-attention mechanisms. In parallel,associative memory (AM) frameworks map representations onto energy landscapes, offering interpretable retrieval mechanisms. However, their continuous-time inference dynamics lack the biological plausibility of classical Continuous Attractor Neural Networks (CANNs). To bridge this gap, we propose Controlled Dynamics Attractor Transformer (CDAT), which couples a mixture von Mises-Fisher (Mo-vMF) attention energy with a Hopfield refinement energy, while augmenting energy descent with a CANN-inspired excitation-inhibition modulation. CDAT instantiates a topology-constrained dynamical system whose couplings encode relational structure among tokens, thereby linking attractor-style dynamics to modern energy-based attention. We further provide a constructive dissipation analysis to formally establish their controlled inference dynamics. Benefiting from these robust and structured dynamics, CDAT achieves state-of-the-art performance across multiple benchmarks in graph anomaly detection and graph classification.

Problem

• attractor dynamics
• energy-based attention
• biological plausibility
• Continuous Attractor Neural Networks
• representation learning

Innovation

• Controlled Dynamics Attractor Transformer
• energy-based attention
• Continuous Attractor Neural Networks
• mixture von Mises-Fisher
• topology-constrained dynamical system