Convergence Analysis of Real-time Recurrent Learning (RTRL) for a class of Recurrent Neural Networks

2025-01-14

AI Summary

This paper addresses the training bias and instability arising from gradient approximation in truncated backpropagation through time (BPTT) for RNNs modeling long sequences. It systematically investigates the theoretical foundations and practical performance of real-time recurrent learning (RTRL) in financial time series modeling. First, it establishes, for the first time under infinite-horizon settings, the convergence of RTRL to stationary points of the loss function, via a fixed-point theoretical framework integrating data distribution, hidden-state dynamics, and forward-mode derivative propagation. Second, it analyzes RTRL’s online optimization behavior using tools from stochastic processes and nonlinear dynamical systems. Empirical evaluation on long-range financial time series—such as limit order book data—demonstrates that RTRL significantly outperforms truncated BPTT in both prediction accuracy and training stability, while remaining computationally feasible for small-to-medium-scale RNNs.

Technology Category

• Machine Learning: Online Learning & Bandits
• Natural Language Processing: Learning & Optimization for NLP
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Recurrent neural networks (RNNs) are commonly trained with the truncated backpropagation-through-time (TBPTT) algorithm. For the purposes of computational tractability, the TBPTT algorithm truncates the chain rule and calculates the gradient on a finite block of the overall data sequence. Such approximation could lead to significant inaccuracies, as the block length for the truncated backpropagation is typically limited to be much smaller than the overall sequence length. In contrast, Real-time recurrent learning (RTRL) is an online optimization algorithm which asymptotically follows the true gradient of the loss on the data sequence as the number of sequence time steps $t ightarrow infty$. RTRL forward propagates the derivatives of the RNN hidden/memory units with respect to the parameters and, using the forward derivatives, performs online updates of the parameters at each time step in the data sequence. RTRL's online forward propagation allows for exact optimization over extremely long data sequences, although it can be computationally costly for models with large numbers of parameters. We prove convergence of the RTRL algorithm for a class of RNNs. The convergence analysis establishes a fixed point for the joint distribution of the data sequence, RNN hidden layer, and the RNN hidden layer forward derivatives as the number of data samples from the sequence and the number of training steps tend to infinity. We prove convergence of the RTRL algorithm to a stationary point of the loss. Numerical studies illustrate our theoretical results. One potential application area for RTRL is the analysis of financial data, which typically involve long time series and models with small to medium numbers of parameters. This makes RTRL computationally tractable and a potentially appealing optimization method for training models. Thus, we include an example of RTRL applied to limit order book data.

Problem

• Real-Time Recursive Learning
• Recurrent Neural Networks
• Financial Data Analysis

Innovation

• Real-Time Recursive Learning
• Long Sequence Data
• Financial Data Analysis