Flow Equivariant Recurrent Neural Networks

2025-07-19

AI Summary

Existing equivariant neural networks support only static transformations and feedforward architectures, rendering them inadequate for modeling geometric symmetries that evolve continuously over time in sequential data—such as visual motion. This work introduces *flow equivariance*—a continuous-time symmetry formalized via one-parameter Lie group actions—into recurrent neural networks, proposing the Flow-Equivariant RNN (FE-RNN). By leveraging differential geometry to model the co-evolution of hidden states on manifolds driven by inputs, FE-RNN enforces strict equivariance of hidden states under time-parameterized transformations. This breaks the traditional limitation of equivariant networks to static data. Empirically, FE-RNN achieves significantly improved training efficiency on next-step prediction and sequence classification tasks, while demonstrating strong generalization to unseen sequence lengths and motion speeds.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Data arrives at our senses as a continuous stream, smoothly transforming from one instant to the next. These smooth transformations can be viewed as continuous symmetries of the environment that we inhabit, defining equivalence relations between stimuli over time. In machine learning, neural network architectures that respect symmetries of their data are called equivariant and have provable benefits in terms of generalization ability and sample efficiency. To date, however, equivariance has been considered only for static transformations and feed-forward networks, limiting its applicability to sequence models, such as recurrent neural networks (RNNs), and corresponding time-parameterized sequence transformations. In this work, we extend equivariant network theory to this regime of `flows' -- one-parameter Lie subgroups capturing natural transformations over time, such as visual motion. We begin by showing that standard RNNs are generally not flow equivariant: their hidden states fail to transform in a geometrically structured manner for moving stimuli. We then show how flow equivariance can be introduced, and demonstrate that these models significantly outperform their non-equivariant counterparts in terms of training speed, length generalization, and velocity generalization, on both next step prediction and sequence classification. We present this work as a first step towards building sequence models that respect the time-parameterized symmetries which govern the world around us.

Problem

• Extend equivariant networks to handle continuous time-parameterized transformations
• Address lack of flow equivariance in standard recurrent neural networks
• Improve sequence model performance on temporal generalization tasks

Innovation

• Extends equivariant networks to continuous time flows
• Introduces flow equivariance in recurrent neural networks
• Improves training speed and generalization performance