Unlocking the Potential of Soft Actor-Critic for Imitation Learning

2025-09-29

AI Summary

Existing imitation learning (IL)-based robot motion generation methods rely heavily on proximal policy optimization (PPO), suffering from low sample efficiency and poor generalization. To address this, we propose AMP-SAC—the first integration of the off-policy Soft Actor-Critic (SAC) algorithm into the Adversarial Motion Priors (AMP) framework. AMP-SAC incorporates entropy-regularized exploration and experience replay, jointly improving data utilization efficiency, policy robustness, and motion naturalness. Experiments across diverse locomotion modes and complex terrains demonstrate that AMP-SAC achieves a +12.7% improvement in imitation reward over AMP+PPO while maintaining stable task execution. These results validate AMP-SAC’s synergistic gains in three key dimensions: sample efficiency, cross-terrain generalization, and motion fidelity.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Motion and Path Planning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Learning-based methods have enabled robots to acquire bio-inspired movements with increasing levels of naturalness and adaptability. Among these, Imitation Learning (IL) has proven effective in transferring complex motion patterns from animals to robotic systems. However, current state-of-the-art frameworks predominantly rely on Proximal Policy Optimization (PPO), an on-policy algorithm that prioritizes stability over sample efficiency and policy generalization. This paper proposes a novel IL framework that combines Adversarial Motion Priors (AMP) with the off-policy Soft Actor-Critic (SAC) algorithm to overcome these limitations. This integration leverages replay-driven learning and entropy-regularized exploration, enabling naturalistic behavior and task execution, improving data efficiency and robustness. We evaluate the proposed approach (AMP+SAC) on quadruped gaits involving multiple reference motions and diverse terrains. Experimental results demonstrate that the proposed framework not only maintains stable task execution but also achieves higher imitation rewards compared to the widely used AMP+PPO method. These findings highlight the potential of an off-policy IL formulation for advancing motion generation in robotics.

Problem

• Improving sample efficiency in imitation learning algorithms
• Enhancing policy generalization for robotic motion control
• Overcoming limitations of on-policy methods in motion imitation

Innovation

• Combines Adversarial Motion Priors with Soft Actor-Critic
• Leverages replay-driven learning and entropy-regularized exploration
• Enables naturalistic behavior and improves data efficiency