ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints

2026-04-16

AI Summary

Existing embodied agents often neglect the manipulability of target objects during instruction execution, lacking the ability to perceive and reason about implicit affordance constraints. To address this limitation, this work introduces DynAfford, a dynamic affordance benchmark that, for the first time, evaluates agents’ commonsense planning capabilities in scenarios where affordance constraints are not explicitly specified. Furthermore, the authors propose ADAPT, a plug-and-play module that integrates a vision-language model—fine-tuned with LoRA and adapted to the domain—as an affordance reasoning backend, thereby endowing existing planners with explicit, task-adaptive affordance judgment. Experimental results demonstrate that ADAPT significantly improves task success rates and robustness in both seen and unseen environments, outperforming GPT-4o in affordance-aware planning.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Manipulation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Intelligent embodied agents should not simply follow instructions, as real-world environments often involve unexpected conditions and exceptions. However, existing methods usually focus on directly executing instructions, without considering whether the target objects can actually be manipulated, meaning they fail to assess available affordances. To address this limitation, we introduce DynAfford, a benchmark that evaluates embodied agents in dynamic environments where object affordances may change over time and are not specified in the instruction. DynAfford requires agents to perceive object states, infer implicit preconditions, and adapt their actions accordingly. To enable this capability, we introduce ADAPT, a plug-and-play module that augments existing planners with explicit affordance reasoning. Experiments demonstrate that incorporating ADAPT significantly improves robustness and task success across both seen and unseen environments. We also show that a domain-adapted, LoRA-finetuned vision-language model used as the affordance inference backend outperforms a commercial LLM (GPT-4o), highlighting the importance of task-aligned affordance grounding.

Problem

• commonsense planning
• affordance constraints
• embodied agents
• dynamic environments
• instruction following

Innovation

• affordance reasoning
• embodied AI
• dynamic environments
• plug-and-play module
• vision-language model