ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network

2026-05-10

AI Summary

This work addresses the challenge in long-horizon, sparse-reward reinforcement learning where fixed action chunking struggles to balance responsiveness and temporal consistency. The authors propose an adaptive action chunking method within an Actor-Critic framework, leveraging a causal Transformer-based Q-network that dynamically selects state-dependent optimal chunk lengths at each boundary. This approach introduces, for the first time, a task-agnostic adaptive mechanism that requires no environment-specific hyperparameter tuning. Theoretical analysis establishes that the associated Bellman operator possesses a unique fixed point, guaranteeing policy convergence. The proposed framework enables multi-scale value estimation over variable-length action chunks and achieves state-of-the-art performance on the OGBench benchmark under both offline and offline-to-online settings.

Abstract

Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actions, which reduce the effective horizon, enable fast value backups, and support temporally consistent exploration. However, existing methods rely on a fixed chunk size and therefore cannot adaptively balance reactivity against temporal consistency. A large fixed chunk size reduces responsiveness to new observations, while a small one produces incoherent motions, forcing task-specific tuning of the chunk size. To address this limitation, we propose Adaptive Chunk Size Actor-Critic (ACSAC). ACSAC leverages a causal Transformer critic to evaluate expected returns for action chunks of different sizes. At each chunk boundary, it adaptively selects the chunk size that maximizes the expected return, supporting flexible, state-dependent chunk sizes without task-specific tuning. We prove that the ACSAC Bellman operator is a contraction whose unique fixed point is the action-value function of the adaptive policy. Experiments on OGBench demonstrate that ACSAC achieves state-of-the-art performance on long-horizon, sparse-reward manipulation tasks across both offline RL and offline-to-online RL settings.

Problem

• long-horizon
• sparse-reward
• action chunking
• adaptive chunk size
• reinforcement learning

Innovation

• Adaptive Chunk Size
• Causal Transformer
• Actor-Critic
• Long-horizon RL
• Sparse-reward Tasks