Kinematics-Aware Multi-Policy Reinforcement Learning for Force-Capable Humanoid Loco-Manipulation

2025-11-26

AI Summary

In high-load industrial scenarios, humanoid robots struggle to simultaneously achieve dexterous manipulation and active physical interaction. To address this, we propose a reinforcement learning–based three-stage decoupled training framework for coordinated upper- and lower-limb force-controlled locomotion and manipulation. Methodologically: (1) forward kinematics priors are implicitly embedded to accelerate convergence of the upper-body policy; (2) an interaction-force–driven curriculum learning mechanism enables the robot to actively exert and dynamically modulate contact forces with the environment; and (3) multi-policy RL, heuristic reward modeling, and whole-body coordination optimization are integrated. Experiments demonstrate that our framework significantly improves force control accuracy and training efficiency, while exhibiting superior robustness and generalization across complex industrial tasks—particularly under varying payload and environmental uncertainty.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Humanoid robots, with their human-like morphology, hold great potential for industrial applications. However, existing loco-manipulation methods primarily focus on dexterous manipulation, falling short of the combined requirements for dexterity and proactive force interaction in high-load industrial scenarios. To bridge this gap, we propose a reinforcement learning-based framework with a decoupled three-stage training pipeline, consisting of an upper-body policy, a lower-body policy, and a delta-command policy. To accelerate upper-body training, a heuristic reward function is designed. By implicitly embedding forward kinematics priors, it enables the policy to converge faster and achieve superior performance. For the lower body, a force-based curriculum learning strategy is developed, enabling the robot to actively exert and regulate interaction forces with the environment.

Problem

• Addresses humanoid robots' limitations in industrial force-capable loco-manipulation tasks
• Overcomes lack of combined dexterity and proactive force interaction capabilities
• Develops kinematics-aware reinforcement learning for high-load industrial manipulation scenarios

Innovation

• Decoupled three-stage reinforcement learning training pipeline
• Heuristic reward function with forward kinematics priors
• Force-based curriculum learning for lower body control