CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization

2026-02-23

AI Summary

This work proposes an efficient continuous Actor-Critic framework to address the high computational cost and poor scalability of integrating trajectory optimization with reinforcement learning in high-dimensional systems. By leveraging properties of the value function under locally optimal policies, the method introduces a sampling strategy biased toward the initial state distribution. Furthermore, it pioneers the integration of GPU-accelerated trajectory optimization into the CACTO framework, substantially improving both data efficiency and computational speed. Experimental results demonstrate that the proposed approach outperforms CACTO and PPO in terms of sample efficiency and convergence rate, and successfully enables real-time high-dimensional control on the AlienGO quadrupedal robot.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Learning & Optimization for ROB
• Constraint Satisfaction and Optimization: Constraint Optimization

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Trajectory Optimization (TO) and Reinforcement Learning (RL) offer complementary strengths for solving optimal control problems. TO efficiently computes locally optimal solutions but can struggle with non-convexity, while RL is more robust to non-convexity at the cost of significantly higher computational demands. CACTO (Continuous Actor-Critic with Trajectory Optimization) was introduced to combine these advantages by learning a warm-start policy that guides the TO solver towards low-cost trajectories. However, scalability remains a key limitation, as increasing system complexity significantly raises the computational cost of TO. This work introduces CACTO-BIC to address these challenges. CACTO-BIC improves data efficiency by biasing initial-state sampling leveraging a property of the value function associated with locally optimal policies; moreover, it reduces computation time by exploiting GPU acceleration. Empirical evaluations show improved sample efficiency and faster computation compared to CACTO. Comparisons with PPO demonstrate that our approach can achieve similar solutions in less time. Finally, experiments on the AlienGO quadruped robot demonstrate that CACTO-BIC can scale to high-dimensional systems and is suitable for real-time applications.

Problem

• Scalability
• Trajectory Optimization
• Reinforcement Learning
• Optimal Control
• High-dimensional Systems

Innovation

• biased sampling
• GPU acceleration
• trajectory optimization
• actor-critic
• scalable reinforcement learning