Intentional Updates for Streaming Reinforcement Learning

2026-04-20

AI Summary

This work addresses the instability and uncontrolled update magnitudes in streaming reinforcement learning caused by fixed step sizes. Inspired by the normalized least mean squares (NLMS) algorithm, we propose a novel approach that adaptively determines step sizes by inversely reasoning from desired update effects. Extending this principle to deep reinforcement learning, we introduce Intentional TD and Intentional Policy Gradient algorithms, which incorporate eligibility traces and diagonal scaling to enable stable and efficient learning in a purely streaming setting—without relying on experience replay buffers. Empirical results demonstrate that our method achieves state-of-the-art streaming performance across multiple benchmark tasks, matching or surpassing batch-based algorithms that depend on experience replay.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Multiagent Learning
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

In gradient-based learning, a step size chosen in parameter units does not produce a predictable per-step change in function output. This often leads to instability in the streaming setting (i.e., batch size=1), where stochasticity is not averaged out and update magnitudes can momentarily become arbitrarily big or small. Instead, we propose intentional updates: first specify the intended outcome of an update and then solve for the step size that approximately achieves it. This strategy has precedent in online supervised linear regression via Normalized Least Mean Squares algorithm, which selects a step size to yield a specified change in the function output proportional to the current error. We extend this principle to streaming deep reinforcement learning by defining appropriate intended outcomes: Intentional TD aims for a fixed fractional reduction of the TD error, and Intentional Policy Gradient aims for a bounded per-step change in the policy, limiting local KL divergence. We propose practical algorithms combining eligibility traces and diagonal scaling. Empirically, these methods yield state-of-the-art streaming performance, frequently performing on par with batch and replay-buffer approaches.

Problem

• streaming reinforcement learning
• gradient-based learning
• update instability
• step size
• function output change

Innovation

• intentional updates
• streaming reinforcement learning
• adaptive step size
• Normalized Least Mean Squares
• eligibility traces