Horizon Reduction Makes RL Scalable

2025-06-04

AI Summary

Offline reinforcement learning (RL) suffers from poor scalability in complex tasks and large-scale datasets, primarily due to excessively long task horizons—this fundamental bottleneck prevents performance from improving with increasing data volume and computational resources. This paper provides the first systematic analysis establishing long horizon as the root cause of offline RL’s non-scalability. To address it, we propose SHARSA, a lightweight and scalable framework that achieves explicit horizon compression via three key mechanisms: trajectory reparameterization, hierarchical state abstraction, and multi-step reward reweighting—fully compatible with mainstream offline RL algorithms such as CQL and IQL. Evaluated on ultra-large-scale offline benchmarks—1,000× larger than standard datasets—SHARSA attains optimal asymptotic performance and near-linear scalability, significantly outperforming existing methods. Our work establishes a new paradigm for scaling offline RL to real-world applications.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: The sharing economy
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In this work, we study the scalability of offline reinforcement learning (RL) algorithms. In principle, a truly scalable offline RL algorithm should be able to solve any given problem, regardless of its complexity, given sufficient data, compute, and model capacity. We investigate if and how current offline RL algorithms match up to this promise on diverse, challenging, previously unsolved tasks, using datasets up to 1000x larger than typical offline RL datasets. We observe that despite scaling up data, many existing offline RL algorithms exhibit poor scaling behavior, saturating well below the maximum performance. We hypothesize that the horizon is the main cause behind the poor scaling of offline RL. We empirically verify this hypothesis through several analysis experiments, showing that long horizons indeed present a fundamental barrier to scaling up offline RL. We then show that various horizon reduction techniques substantially enhance scalability on challenging tasks. Based on our insights, we also introduce a minimal yet scalable method named SHARSA that effectively reduces the horizon. SHARSA achieves the best asymptotic performance and scaling behavior among our evaluation methods, showing that explicitly reducing the horizon unlocks the scalability of offline RL. Code: https://github.com/seohongpark/horizon-reduction

Problem

• Investigates scalability issues in offline RL algorithms
• Identifies long horizons as barrier to offline RL scaling
• Proposes horizon reduction techniques to improve RL scalability

Innovation

• Horizon reduction enhances RL scalability
• SHARSA method effectively reduces horizon
• Long horizons hinder offline RL scaling