CometVLA: Co-Training on an Embodied Data Pyramid towards Physical Understanding

2026-08-31

AI Summary

本文通过构建与机器人动作数据严格对齐的CometData和CometBench，并采用全局动作先验令牌进行协同训练，解决了视觉-语言-动作模型在需要物理常识的操作任务中表现脆弱的问题。

Abstract

Vision-language-action (VLA) models remain brittle in manipulation tasks that require physical commonsense. Current physical VQA data is typically disembodied and misaligned with robot action domains. Egocentric videos are used only as auxiliary pre-training. It remains unclear whether improved VLM physical understanding actually benefits downstream action generation. Therefore, we present CometVLA to close this gap. We construct CometData and CometBench, an embodied physical VQA corpus and benchmark strictly aligned with the robot's action data and embodiment. We introduce Global Action Prior (GAP) tokens, a compact learnable bottleneck that isolates task-agnostic motion regularities and lets the action head consume physical commonsense without corrupting the pre-trained VLM backbone. We co-train CometVLA across the embodied data pyramid, spanning teleoperation, simulation, egocentric trajectories, and VQA layers. On real-world manipulation tasks and RoboTwin simulation, CometVLA consistently outperforms strong VLA baselines. Correlation analysis shows that stronger VLM performance on CometBench indicates higher VLA success rates. Results demonstrate that physical understanding pre-training genuinely benefits downstream manipulation.

Problem

• Vision-language-action
• physical commonsense
• embodied data
• action generation
• manipulation tasks

Innovation

• Co-Training
• Embodied Data Pyramid
• Global Action Prior (GAP)
• Physical Commonsense