Robot Planning and Situation Handling with Active Perception

2026-04-28

AI Summary

This work addresses the challenge of task execution failures in dynamic, open-world environments—such as those caused by jammed doors or unforeseen ground obstacles—by introducing the VAP-TAMP framework. VAP-TAMP uniquely integrates action-knowledge-guided active viewpoint selection with vision-language models and leverages scene graph construction and reasoning to enable joint task and motion planning (TAMP). The proposed approach facilitates real-time detection of and response to execution anomalies, significantly improving both task success rates and robotic autonomy in complex, dynamic settings. Evaluations on both simulated and real-world service robot platforms demonstrate its effectiveness in enhancing robustness and adaptability under uncertainty.

Abstract

Current robots are capable of computing plans to accomplish complex tasks. However, real-world environments are inherently open and dynamic, and unforeseen situations frequently arise during plan execution, such as jamming doors and fallen objects on the floor. These situations may result from the robot's own action failures or from external disturbances, such as human activities. Detecting and handling such execution - time situations remains a significant challenge, limiting those robots' ability to achieve long-term autonomy. In this paper, we develop a planning and situation-handling framework, called VAP-TAMP, that enables robots to actively perceive and address unforeseen situations during plan execution. VAP-TAMP leverages action knowledge to strategically prompt vision-language models for active view selection and situation assessment, while constructing and reasoning over scene graphs for integrated task and motion planning. We evaluated VAP-TAMP using service tasks in simulation and on a mobile manipulation platform.

Problem

• robot planning
• unforeseen situations
• active perception
• execution-time handling
• long-term autonomy

Innovation

• active perception
• vision-language models
• scene graph
• task and motion planning
• situation handling