Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications

2026-05-12

AI Summary

This work addresses the instability and initialization sensitivity of existing ultra-low-power parallelized RNNs—such as BMRU—stemming from gradient blocking, which hinders their ability to handle complex sequential tasks. To overcome these limitations, the authors propose the Cumulative Memory Recurrent Unit (CMRU) and its relaxed variant, αCMRU, which incorporate a cumulative state update mechanism and temporal skip connections. These innovations restore gradient flow across time while preserving quantized states and compatibility with analog hardware. For the first time, this approach unifies stable training and long-range discrete memory capabilities without compromising low power consumption or noise robustness. Experiments demonstrate that CMRU and αCMRU match or surpass the performance of LRU and minGRU in compact models, particularly excelling in tasks requiring long-range memory, with notably improved convergence stability and reduced sensitivity to initialization.

Abstract

Sequence learning is dominated by Transformers and parallelizable recurrent neural networks (RNNs) such as state-space models, yet learning long-term dependencies remains challenging, and state-of-the-art designs trade power consumption for performance. The Bistable Memory Recurrent Unit (BMRU) was introduced to enable hardware-software co-design of ultra-low power RNNs: quantized states with hysteresis provide persistent memory while mapping directly to analog primitives. However, BMRU performance lags behind parallelizable RNNs on complex sequential tasks. In this paper, we identify gradient blocking during state updates as a key limitation and propose a cumulative update formulation that restores gradient flow while preserving persistent memory, creating skip-connections through time. This leads to the Cumulative Memory Recurrent Unit (CMRU) and its relaxed variant, the $α$CMRU. Experiments show that the cumulative formulation dramatically improves convergence stability and reduces initialization sensitivity. The CMRU and $α$CMRU match or outperform Linear Recurrent Units (LRUs) and minimal Gated Recurrent Units (minGRUs) across diverse benchmarks at small model sizes, with particular advantages on tasks requiring discrete long-range retention, while the CMRU retains quantized states, persistent memory, and noise-resilient dynamics essential for analog implementation.

Problem

• parallelizable RNNs
• long-term dependencies
• learning stability
• ultra-low power
• gradient blocking

Innovation

• cumulative update
• persistent memory
• gradient flow
• ultra-low power RNNs
• hardware-software co-design