Self-Attention as a Parametric Endofunctor: A Categorical Framework for Transformer Architectures

2025-01-06

AI Summary

Transformer self-attention lacks a unified mathematical semantics. Method: We propose the first formal framework grounded in category theory: modeling Q/K/V mappings as parameterized endofunctors on the category Param(Vect) of parameterized vector spaces; stacking multiple layers as the free monad generated by such an endofunctor; and interpreting positional encodings either as monoid actions or universal constructions. Contributions: (1) We identify, for the first time, the linear component of self-attention as a parameterized endofunctor; (2) We unify the algebraic structure of positional encodings with the morphism-composition nature of “circuits” in mechanistic interpretability; (3) We establish a rigorous categorical characterization of attention’s permutation equivariance. This framework provides the first endogenous, formally verifiable, and algebraically principled semantic foundation for Transformers.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

Self-attention mechanisms have revolutionised deep learning architectures, but their mathematical foundations remain incompletely understood. We establish that these mechanisms can be formalised through categorical algebra, presenting a framework that focuses on the linear components of self-attention. We prove that the query, key, and value maps in self-attention naturally form a parametric endofunctor in the 2-category $mathbf{Para}(mathbf{Vect})$ of parametric morphisms. We show that stacking multiple self-attention layers corresponds to constructing the free monad on this endofunctor. For positional encodings, we demonstrate that strictly additive position embeddings constitute monoid actions on the embedding space, while standard sinusoidal encodings, though not additive, possess a universal property among faithful position-preserving functors. We establish that the linear portions of self-attention exhibit natural equivariance properties with respect to permutations of input tokens. Finally, we prove that the ``circuits'' identified in mechanistic interpretability correspond precisely to compositions of parametric morphisms in our framework. This categorical perspective unifies geometric, algebraic, and interpretability-based approaches to transformer analysis, while making explicit the mathematical structures underlying attention mechanisms. Our treatment focuses exclusively on linear maps, setting aside nonlinearities like softmax and layer normalisation, which require more sophisticated categorical structures. Our results extend recent work on categorical foundations for deep learning while providing insights into the algebraic structure of attention mechanisms.

Problem

• Transformer Architecture
• Self-Attention Mechanism
• Mathematical Principles

Innovation

• Categorical Algebra
• Self-Attention Mechanism
• Positional Encoding