Equivariant non-linear maps for neural networks on homogeneous spaces

2025-04-29

AI Summary

This work addresses the fundamental challenge of designing nonlinear equivariant neural network layers on homogeneous spaces. We establish, for the first time, a representation theory for nonlinear equivariant layers in this setting. Introducing the generalized steerability constraint, we unify the symmetry mechanisms underlying prominent architectures—including G-CNNs, implicitly steerable kernel networks, positional-embedding attention, and LieTransformers—and derive and prove a universality theorem characterizing all nonlinear mappings satisfying this constraint. Our methodology integrates group representation theory, differential geometry of homogeneous spaces, and nonlinear operator theory to systematically derive and interpret the structural origins of diverse equivariant models. The core contribution is a universal, theoretically grounded framework for nonlinear equivariant layers on homogeneous spaces—providing the first general, verifiable construction principle for symmetry-constrained neural networks. (138 words)

Technology Category

• Machine Learning: Learning with Manifolds
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper presents a novel framework for non-linear equivariant neural network layers on homogeneous spaces. The seminal work of Cohen et al. on equivariant $G$-CNNs on homogeneous spaces characterized the representation theory of such layers in the linear setting, finding that they are given by convolutions with kernels satisfying so-called steerability constraints. Motivated by the empirical success of non-linear layers, such as self-attention or input dependent kernels, we set out to generalize these insights to the non-linear setting. We derive generalized steerability constraints that any such layer needs to satisfy and prove the universality of our construction. The insights gained into the symmetry-constrained functional dependence of equivariant operators on feature maps and group elements informs the design of future equivariant neural network layers. We demonstrate how several common equivariant network architectures - $G$-CNNs, implicit steerable kernel networks, conventional and relative position embedded attention based transformers, and LieTransformers - may be derived from our framework.

Problem

• Generalize equivariant neural networks to non-linear layers
• Derive steerability constraints for non-linear equivariant operators
• Unify diverse architectures under a universal framework

Innovation

• Non-linear equivariant layers on homogeneous spaces
• Generalized steerability constraints for layers
• Universality proof for the proposed construction