Environmental Understanding Vision-Language Model for Embodied Agent

2026-04-21

AI Summary

This work addresses the frequent failures of existing vision-language models (VLMs) in embodied intelligence tasks, which stem from insufficient environmental understanding and overreliance on metadata. The authors propose the Environment Understanding Embodied Agent (EUEA) framework, which systematically defines and fine-tunes four core VLM capabilities: object perception, task planning, action comprehension, and goal recognition. Integrating an action recovery mechanism with Group Relative Policy Optimization (GRPO), EUEA effectively corrects execution errors and enhances prediction consistency. Evaluated on the ALFRED benchmark, EUEA improves average success rates by 8.86% over behavior cloning baselines, with the recovery mechanism and GRPO jointly contributing an additional 3.03% gain. The study also reveals critical limitations of both open- and closed-source VLMs in environmental understanding.

Technology Category

• Intelligent Robots: Embodied AI
• Computer Vision: Language and Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Vision-language models (VLMs) have shown strong perception and reasoning abilities for instruction-following embodied agents. However, despite these abilities and their generalization performance, they still face limitations in environmental understanding, often failing on interactions or relying on environment metadata during execution. To address this challenge, we propose a novel framework named Environmental Understanding Embodied Agent (EUEA), which fine-tunes four core skills: 1) object perception for identifying relevant objects, 2) task planning for generating interaction subgoals, 3) action understanding for judging success likelihood, and 4) goal recognition for determining goal completion. By fine-tuning VLMs with EUEA skills, our framework enables more reliable task execution for instruction-following. We further introduce a recovery step that leverages these core skills and a group relative policy optimization (GRPO) stage that refines inconsistent skill predictions. The recovery step samples alternative actions to correct failure cases, and the GRPO stage refines inconsistent skill predictions. Across ALFRED tasks, our VLM significantly outperforms a behavior-cloning baseline, achieving an 8.86% improvement in average success rate. The recovery and GRPO stages provide an additional 3.03% gain, further enhancing overall performance. Finally, our skill-level analyses reveal key limitations in the environmental understanding of closed- and open-source VLMs and identify the capabilities necessary for effective agent-environment interaction.

Problem

• environmental understanding
• vision-language models
• embodied agents
• instruction-following
• task execution

Innovation

• Environmental Understanding
• Vision-Language Model
• Embodied Agent
• Recovery Mechanism
• Group Relative Policy Optimization