Direction Matters: Learning Force Direction Enables Sim-to-Real Contact-Rich Manipulation

2026-02-15

AI Summary

This work addresses the performance degradation in sim-to-real transfer for contact-intensive manipulation tasks caused by discrepancies in contact dynamics between simulation and reality. The authors propose a novel approach that centers on learning the direction of contact forces as a key objective. In simulation, a privileged supervisory signal is provided by an expert-designed state-machine-based position/force controller to train a policy that predicts end-effector pose, contact state, and force direction. At deployment, a lightweight fine-tuned constant force magnitude is combined with the predicted force direction to drive a force-aware admittance controller, enabling adaptive compliant control. Force direction is robust to simulation inaccuracies and encodes rich task-relevant geometric semantics, allowing effective transfer with only a single scalar parameter. Experiments on four real-world tasks—microwave door opening, peg-in-hole assembly, whiteboard wiping, and door opening—demonstrate significantly higher success rates and robustness compared to strong baselines.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Sim-to-real transfer for contact-rich manipulation remains challenging due to the inherent discrepancy in contact dynamics. While existing methods often rely on costly real-world data or utilize blind compliance through fixed controllers, we propose a framework that leverages expert-designed controller logic for transfer. Inspired by the success of privileged supervision in kinematic tasks, we employ a human-designed finite state machine based position/force controller in simulation to provide privileged guidance. The resulting policy is trained to predict the end-effector pose, contact state, and crucially the desired contact force direction. Unlike force magnitudes, which are highly sensitive to simulation inaccuracies, force directions encode high-level task geometry and remain robust across the sim-to-real gap. At deployment, these predictions configure a force-aware admittance controller. By combining the policy's directional intent with a constant, low-cost manually tuned force magnitude, the system generates adaptive, task-aligned compliance. This tuning is lightweight, typically requiring only a single scalar per contact state. We provide theoretical analysis for stability and robustness to disturbances. Experiments on four real-world tasks, i.e., microwave opening, peg-in-hole, whiteboard wiping, and door opening, demonstrate that our approach significantly outperforms strong baselines in both success rate and robustness. Videos are available at: https://yifei-y.github.io/project-pages/DirectionMatters/.

Problem

• sim-to-real transfer
• contact-rich manipulation
• contact dynamics
• force direction
• robotic manipulation

Innovation

• force direction
• sim-to-real transfer
• contact-rich manipulation
• admittance control
• privileged supervision