Effective Task Planning with Missing Objects using Learning-Informed Object Search

2026-02-12

AI Summary

This work addresses the challenge of task planning under partial observability, where traditional planners fail when key object locations are unknown. The authors propose a novel planning framework that models learning-driven object search as high-level deterministic actions—Learned Interactive Object Search (LIOS)—each encapsulating a policy to locate and retrieve a specific object. By integrating LIOS into classical PDDL-based task planning through model-based expected cost estimation, the approach explicitly accounts for environmental uncertainty while remaining compatible with existing omniscient solvers. Evaluated on object retrieval and meal preparation tasks in both ProcTHOR simulation and real-world settings, the method significantly outperforms both non-learning and learning baselines, marking the first end-to-end integration of learned search strategies with symbolic task planning.

Technology Category

• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Task planning for mobile robots often assumes full environment knowledge and so popular approaches, like planning via the PDDL, cannot plan when the locations of task-critical objects are unknown. Recent learning-driven object search approaches are effective, but operate as standalone tools and so are not straightforwardly incorporated into full task planners, which must additionally determine both what objects are necessary and when in the plan they should be sought out. To address this limitation, we develop a planning framework centered around novel model-based LIOS actions: each a policy that aims to find and retrieve a single object. High-level planning treats LIOS actions as deterministic and so -- informed by model-based calculations of the expected cost of each -- generates plans that interleave search and execution for effective, sound, and complete learning-informed task planning despite uncertainty. Our work effectively reasons about uncertainty while maintaining compatibility with existing full-knowledge solvers. In simulated ProcTHOR homes and in the real world, our approach outperforms non-learned and learned baselines on tasks including retrieval and meal prep.

Problem

• task planning
• missing objects
• object search
• mobile robots
• environment uncertainty

Innovation

• learning-informed planning
• object search
• task planning under uncertainty
• model-based action
• mobile robot planning