RePlan-Bot: Multi-Level Replanning for Embodied Instruction Following

2026-05-25

AI Summary

This work addresses the low success rates in embodied instruction-following tasks caused by challenges in long-horizon planning and irreversible state changes. To this end, the authors propose RePlan-Bot, which introduces, for the first time, a multi-level replanning mechanism that operates throughout the entire execution process. The approach integrates a large language model–driven high-level auditor, a commonsense-guided multi-layer instance map search, and a lightweight ViT-based action correction module. This enables dynamic subgoal adjustment driven by environmental feedback, structured object localization, and preemptive risk-aware action refinement. Evaluated on the ALFRED benchmark, RePlan-Bot achieves state-of-the-art performance, significantly improving task success rates in both seen and unseen environments.

Abstract

Embodied instruction following (EIF) requires agents to understand and execute complex natural language commands within interactive 3D environments. Despite recent advances, existing methods often fail in long-horizon planning and handling irreversible state changes, resulting in low task success rates. To address these challenges, we introduce RePlan-Bot, a novel EIF agent that performs multi-level, continuous replanning throughout task execution. RePlan-Bot integrates a high-level LLM-based auditor for dynamic sub-goal adjustments guided by environmental feedback, a commonsense-guided search mechanism based on a multi-layered instance map for precise and structured object localization, and a lightweight ViT-based corrector to preemptively fix risky low-level actions. Evaluated on the ALFRED benchmark, RePlan-Bot achieves state-of-the-art performance in both seen and unseen environments, demonstrating superior adaptability and reliability.

Problem

• Embodied Instruction Following
• Long-horizon Planning
• Irreversible State Changes
• Task Success Rate

Innovation

• multi-level replanning
• LLM-based auditor
• commonsense-guided search
• instance map
• ViT-based corrector