Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge for Controllable Dexterous World Models

2026-07-05

AI Summary

Existing monolithic world models struggle to generate fine-grained, controllable video predictions for dexterous hands with 23 degrees of freedom. To address this challenge, this work proposes a two-stage action-conditioned world model: it first predicts a sequence of segmentation masks using a dynamics model, then renders these masks into photorealistic RGB videos via a ControlNet-enhanced Stable Video Diffusion pipeline. By introducing segmentation masks as an intermediate representation bridging simulation and reality, the approach decouples dynamics modeling from image rendering, enabling precise control over all degrees of freedom. Leveraging simulation-based pretraining followed by fine-tuning on only 2.5 hours of real-world data, the method significantly outperforms end-to-end baselines on dexterous grasping tasks.

Abstract

Action-conditioned world models allow robots to predict the future consequences of candidate actions without additional physical interaction, supporting policy evaluation, planning, and data augmentation. We present Mask2Real-WM, a two-stage action-conditioned world model for dexterous manipulation that decouples pixel prediction into a dynamics model and a rendering model. The dynamics model predicts future segmentation masks from past masks and 23-DoF action sequences. The rendering model maps the predicted masks to photorealistic RGB using a ControlNet-augmented Stable Video Diffusion backbone. The smaller sim-to-real gap in segmentation space enables the dynamics model to benefit from large-scale pretraining on over 50 h of synthetic simulation data, followed by fine-tuning on fewer than 2.5 h of real demonstrations. Experiments on a dexterous pick-and-place benchmark show that mask conditioning and simulation pretraining are both required for per-DoF action controllability across all 23 degrees of freedom. In contrast, monolithic baselines capture broad hand and end-effector trajectories but do not reliably reflect fine-grained, per-joint action effects.

Problem

• dexterous manipulation
• world models
• sim-to-real
• action-conditioned prediction
• segmentation masks

Innovation

• world model
• sim-to-real
• segmentation mask
• dexterous manipulation
• action-conditioned prediction