Modality-Autoregressive World-Action Models

2026-09-15

AI Summary

本文提出ModAR模型，通过自回归去噪多种未来模态来预测动作，以解决如何在WAMs中有效结合不同视觉模态的问题。

Abstract

World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions. This allows each prediction to condition on previously generated modalities. We train from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance. In our evaluations, WAMs benefit from predicting point tracks, DINO features, and depth maps, while additionally predicting future RGB does not provide a consistent benefit. We also find that ModAR's sequential generation outperforms existing WAM formulations, with the highest average success rate at all evaluated data scales. We also fine-tune the video-model-initialized WAM Flex-$π$ on the same data; ModAR achieves a slightly higher observed average success rate (75% vs. 72%) while using approximately $20\times$ fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR outperforms baselines and improves with human videos.

Problem

• World-Action Models
• Visual Modalities
• Geometric Features
• Semantic Features
• Motion Features

Innovation

• ModAR
• autoregressive denoising
• multiple future modalities
• sequential generation
• performance improvement