Platonic Transformers: A Solid Choice For Equivariance

2025-10-03

AI Summary

Standard Transformers lack inductive biases for geometric symmetries, and existing equivariant approaches often compromise computational efficiency or architectural flexibility. Method: We propose the Platonic Transformer—the first architecture to jointly enforce strict equivariance to continuous translations and discrete Platonic symmetries (e.g., tetrahedral, octahedral) within the standard Transformer framework. Our approach rests on three key innovations: (1) establishing theoretical equivalence between self-attention and dynamic group convolution; (2) designing relative attention and weight-sharing mechanisms grounded in Platonic symmetry-group reference frames; and (3) deriving a linear-time equivariant convolution variant. Contribution/Results: The method requires no architectural modification or additional computational overhead, yet achieves significant performance gains across diverse benchmarks—including CIFAR-10, ScanObjectNN, QM9, and OMol25—demonstrating strong equivariance, broad applicability, and high efficiency.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

While widespread, Transformers lack inductive biases for geometric symmetries common in science and computer vision. Existing equivariant methods often sacrifice the efficiency and flexibility that make Transformers so effective through complex, computationally intensive designs. We introduce the Platonic Transformer to resolve this trade-off. By defining attention relative to reference frames from the Platonic solid symmetry groups, our method induces a principled weight-sharing scheme. This enables combined equivariance to continuous translations and Platonic symmetries, while preserving the exact architecture and computational cost of a standard Transformer. Furthermore, we show that this attention is formally equivalent to a dynamic group convolution, which reveals that the model learns adaptive geometric filters and enables a highly scalable, linear-time convolutional variant. Across diverse benchmarks in computer vision (CIFAR-10), 3D point clouds (ScanObjectNN), and molecular property prediction (QM9, OMol25), the Platonic Transformer achieves competitive performance by leveraging these geometric constraints at no additional cost.

Problem

• Transformers lack geometric symmetry biases for scientific applications
• Existing equivariant methods sacrifice Transformer efficiency and flexibility
• Platonic Transformer enables geometric equivariance without architectural changes

Innovation

• Attention relative to Platonic solid reference frames
• Preserves standard Transformer architecture and cost
• Equivalent to dynamic group convolution for scalability