๐ค AI Summary
This work proposes the Graph Memory Transformer (GMT), which replaces the feed-forward networks (FFNs) in a standard autoregressive Transformer decoder with an explicit, learnable memory graph while preserving the causal self-attention mechanism. GMT implements interpretable state transitions through a routing-and-displacement scheme based on 128 memory centroids, a 128ร128 directed transition matrix, gravity-source routing, token-conditioned target selection, and gated readout, yielding a pure decoder language model without FFNs. With 82.2M parameters, GMT trains stably and achieves validation loss and perplexity slightly behind those of a 103.0M-parameter GPT baseline (3.5995/36.58 vs. 3.2903/26.85), yet demonstrates comparable zero-shot performance, thereby validating the feasibility and interpretability of the graph-based memory mechanism.
๐ Abstract
We investigate whether the Feed-Forward Network (FFN) sublayer in a decoder-only transformer can be replaced by an explicit learned memory graph while preserving the surrounding autoregressive architecture. The proposed Graph Memory Transformer (GMT) keeps causal self-attention intact, but replaces the usual per-token FFN transformation with a memory cell that routes token representations over a learned bank of centroids connected by a learned directed transition matrix. In the base GMT v7 instantiation studied here, each of 16 transformer blocks contains 128 centroids, a 128 * 128 edge matrix, gravitational source routing, token-conditioned target selection, and a gated displacement readout. The cell therefore returns movement from an estimated source memory state toward a target memory state, rather than a retrieved value. The resulting model is a fully decoder-only language model with 82.2M trainable parameters and no dense FFN sublayers, compared with a 103.0M-parameter dense GPT-style baseline used in the evaluation. The base v7 model trains stably and exposes centroid usage, transition structure, and source-to-target movement as directly inspectable quantities of the forward computation. It remains behind the larger dense baseline in validation loss and perplexity (3.5995/36.58 vs. 3.2903/26.85), while showing close zero-shot benchmark behavior under the evaluated setting. These results are not intended as a state-of-the-art claim; they support the viability and structural interpretability of replacing dense within-token transformation with graph-mediated memory navigation. Broader scaling, optimized kernels, and more extensive benchmark evaluation are left for subsequent work.