Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits

2026-07-08

AI Summary

This work investigates how small Transformer models learn modular integer multiplication under composite moduli, where the operation is not globally invertible. To this end, the authors introduce a “monoid extension” framework that partitions the input space into local algebraic regions, each preserving a group-like structure and incorporating a localized Fourier mechanism. This approach constitutes the first extension of representation-theoretic Fourier analysis to non-group algebraic structures containing zero divisors. Through a combination of representation theory, attention routing analysis, and identification of low-rank writing directions, the study reveals that in square-free modulus multiplication tasks, embeddings are organized by algebraic region, attention exhibits class-sensitive routing, and local Fourier features account for the majority of output logits.

Abstract

Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition. In this work, we investigate how small transformers learn modular integer multiplication over composite moduli, a fundamentally non-invertible operation due to the presence of zero-divisors. We propose the monoid extension: a localized generalization of Group Composition via Representation (GCR) that suggests the learned computation does not rely on a single global representation space. Instead, the model partitions the input space into local hierarchical algebraic regions, where group-like structure survives and Fourier mechanisms can be applied. In transformers trained on square-free modular multiplication, we find that embeddings organize around these regions, attention exhibits class-sensitive routing and low-rank write directions, and local character features explain a large fraction of the model's output logits. Our results suggest that representation-theoretic mechanisms previously identified for group operations can extend beyond groups to more general structures.

Problem

• modular multiplication
• zero-divisors
• non-invertible operation
• monoid
• algebraic structure

Innovation

• monoid extension
• stratified Fourier mechanisms
• modular multiplication
• representation theory
• transformer circuits