Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns

2025-03-05

AI Summary

To address the high variance and training instability induced by *n*-step returns in Soft Actor-Critic (SAC), and the coarse value estimation arising from action chunking applied solely to the actor, this paper proposes **Chunked-TSAC**: a novel paradigm that explicitly encodes and feeds action chunks into the critic network. Crucially, it is the first to integrate Transformer-based sequence modeling with *n*-step temporal-difference returns—enabling low-variance, highly robust value learning without importance sampling. The core innovation lies in migrating action chunking from the actor to the critic, empowering the critic to jointly model intra-chunk temporal dependencies and inter-chunk long-horizon returns. Experiments demonstrate that Chunked-TSAC significantly outperforms standard SAC and leading variants on sparse-reward and multi-stage tasks, achieving faster convergence, enhanced training stability, and superior final policy performance.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Algorithmic accountability and transparency on the web
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Soft Actor-Critic (SAC) critically depends on its critic network, which typically evaluates a single state-action pair to guide policy updates. Using N-step returns is a common practice to reduce the bias in the target values of the critic. However, using N-step returns can again introduce high variance and necessitates importance sampling, often destabilizing training. Recent algorithms have also explored action chunking-such as direct action repetition and movement primitives-to enhance exploration. In this paper, we propose a Transformer-based Critic Network for SAC that integrates the N-returns framework in a stable and efficient manner. Unlike approaches that perform chunking in the actor network, we feed chunked actions into the critic network to explore potential performance gains. Our architecture leverages the Transformer's ability to process sequential information, facilitating more robust value estimation. Empirical results show that this method not only achieves efficient, stable training but also excels in sparse reward/multi-phase environments-traditionally a challenge for step-based methods. These findings underscore the promise of combining Transformer-based critics with N-returns to advance reinforcement learning performance

Problem

• Reduces bias and variance in critic network evaluations.
• Integrates N-step returns for stable and efficient training.
• Enhances performance in sparse reward and multi-phase environments.

Innovation

• Transformer-based Critic Network for SAC
• Integrates N-step returns stably and efficiently
• Feeds chunked actions into critic network