Topoformer: brain-like topographic organization in Transformer language models through spatial querying and reweighting

2025-10-21

AI Summary

Conventional Transformer models lack brain-like spatial topological structure in their hidden representations, resulting in poor interpretability. Method: We propose Topoformer—the first standard Transformer variant to incorporate spatial query mechanisms and local connectivity reweighting, enabling multi-scale topological representation learning over a 2D grid-structured key-value space. Our approach integrates local pooling attention, spatially aware reweighting, and masked language modeling, balancing neurobiological plausibility with computational efficiency. Contribution/Results: Topoformer matches baseline performance on sentiment classification and language modeling tasks; significantly outperforms controls across eight interpretability benchmarks; and exhibits statistically significant spatial alignment between its hidden-layer activation patterns and human fMRI language responses. This work establishes a novel paradigm for developing interpretable, biologically inspired language models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Spatial functional organization is a hallmark of biological brains: neurons are arranged topographically according to their response properties, at multiple scales. In contrast, representations within most machine learning models lack spatial biases, instead manifesting as disorganized vector spaces that are difficult to visualize and interpret. Here, we propose a novel form of self-attention that turns Transformers into"Topoformers"with topographic organization. We introduce spatial querying - where keys and queries are arranged on 2D grids, and local pools of queries are associated with a given key - and spatial reweighting, where we convert the standard fully connected layer of self-attention into a locally connected layer. We first demonstrate the feasibility of our approach by training a 1-layer Topoformer on a sentiment classification task. Training with spatial querying encourages topographic organization in the queries and keys, and spatial reweighting separately encourages topographic organization in the values and self-attention outputs. We then apply the Topoformer motifs at scale, training a BERT architecture with a masked language modeling objective. We find that the topographic variant performs on par with a non-topographic control model on NLP benchmarks, yet produces interpretable topographic organization as evaluated via eight linguistic test suites. Finally, analyzing an fMRI dataset of human brain responses to a large set of naturalistic sentences, we demonstrate alignment between low-dimensional topographic variability in the Topoformer model and human brain language network. Scaling up Topoformers further holds promise for greater interpretability in NLP research, and for more accurate models of the organization of linguistic information in the human brain.

Problem

• Creating brain-like topographic organization in Transformer language models
• Improving model interpretability through spatial querying and reweighting mechanisms
• Aligning language model representations with human brain organization patterns

Innovation

• Spatial querying arranges keys and queries on 2D grids
• Spatial reweighting converts self-attention to locally connected layer
• Topographic organization improves interpretability while maintaining performance