Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation

2026-07-15

AI Summary

This work addresses the challenges faced by real-world embodied agents—such as active exploration, visual distractions, and disambiguating user intent—where existing approaches often rely on privileged simulator information or complete task instructions, limiting their generalization. The authors propose REAL, a framework that achieves, for the first time, an end-to-end deployable agent without requiring prior perceptual assumptions. REAL leverages simulation-to-reality consistent environment APIs, human-in-the-loop simulated user interactions, hierarchical training, and online reinforcement learning to systematically enhance mobile manipulation capabilities in open-world settings. The accompanying REAL-Bench benchmark encompasses 241 tasks, and evaluations on a real dual-arm robot demonstrate a 78.3% end-to-end success rate, with interactive task performance (56.9%) surpassing that of leading commercial closed-source vision-language models, while also exhibiting zero-shot transfer to unseen household environments.

Abstract

Real-world deployment of embodied agents requires active exploration, visual grounding, and interactive intent disambiguation. However, existing frameworks often rely on privileged simulator states or assume complete instructions, bypassing realistic deployment challenges. To bridge this gap, we present REAL, an agentic framework for open-world mobile manipulation. REAL establishes sim-to-real-consistent environment APIs without oracle perception and integrates a simulated user to enable human-in-the-loop interaction. Within this environment, we design diverse task compositions to drive data collection, supervised fine-tuning, and online reinforcement learning, systematically optimizing agent performance. To comprehensively evaluate this approach, we introduce REAL-Bench, a benchmark spanning 241 tasks across active exploration, visual distraction, articulated manipulation, and interactive disambiguation. Experimental results demonstrate that our trained agent outperforms leading commercial closed-source VLMs on interactive tasks with a 56.9% success rate. Further empirical analysis reveals that our hierarchical training pipeline successfully aligns the model's tool-use capabilities while maintaining robust open-vocabulary reasoning under extended exploration horizons. Finally, we deploy and evaluate our framework on a physical dual-arm mobile robot, where it achieves a 78.3% end-to-end success rate over 60 real-world episodes. These physical trials demonstrate robust zero-shot transferability to unseen household scenarios, validating that our sim-to-real-consistent design successfully bridges the reality gap for long-horizon mobile manipulation. Code is available at https://github.com/InternRobotics/REAL.

Problem

• embodied agents
• open-world mobile manipulation
• sim-to-real transfer
• interactive disambiguation
• active exploration

Innovation

• sim-to-real transfer
• embodied agents
• interactive disambiguation
• mobile manipulation
• open-world reasoning