Detecting Invariant Manifolds in ReLU-Based RNNs

2025-10-04

AI Summary

This study investigates the interpretability mechanisms underlying the intrinsic dynamics of ReLU-based recurrent neural networks (RNNs), specifically focusing on the geometric origins of multistability and chaos in their state-space representations. Method: We propose the first algorithm for detecting stable and unstable manifolds in piecewise-linear RNNs (PLRNNs), integrating periodic point analysis, piecewise-linear systems theory, and numerical manifold tracing to precisely characterize attractor boundaries and identify homoclinic points. Contribution/Results: We provide the first rigorous proof of chaotic dynamics in PLRNNs, demonstrating that chaos arises from fractal intersections between stable and unstable manifolds. The method is validated on cortical neuronal electrophysiological data, successfully uncovering multistable structures and elucidating latent dynamical organization principles. This work establishes a novel geometric analytical framework for scientific modeling and interpretable AI.

Technology Category

• Machine Learning: Learning with Manifolds
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Recurrent Neural Networks (RNNs) have found widespread applications in machine learning for time series prediction and dynamical systems reconstruction, and experienced a recent renaissance with improved training algorithms and architectural designs. Understanding why and how trained RNNs produce their behavior is important for scientific and medical applications, and explainable AI more generally. An RNN's dynamical repertoire depends on the topological and geometrical properties of its state space. Stable and unstable manifolds of periodic points play a particularly important role: They dissect a dynamical system's state space into different basins of attraction, and their intersections lead to chaotic dynamics with fractal geometry. Here we introduce a novel algorithm for detecting these manifolds, with a focus on piecewise-linear RNNs (PLRNNs) employing rectified linear units (ReLUs) as their activation function. We demonstrate how the algorithm can be used to trace the boundaries between different basins of attraction, and hence to characterize multistability, a computationally important property. We further show its utility in finding so-called homoclinic points, the intersections between stable and unstable manifolds, and thus establish the existence of chaos in PLRNNs. Finally we show for an empirical example, electrophysiological recordings from a cortical neuron, how insights into the underlying dynamics could be gained through our method.

Problem

• Detecting invariant manifolds in ReLU-based recurrent neural networks
• Tracing boundaries between different basins of attraction
• Establishing existence of chaos through homoclinic points

Innovation

• Novel algorithm detects invariant manifolds in ReLU networks
• Method traces basin boundaries to characterize multistability
• Identifies homoclinic points establishing chaos in PLRNNs