Object Search in Partially-Known Environments via LLM-informed Model-based Planning and Prompt Selection

2026-03-24

AI Summary

This work addresses the challenge of efficiently searching for target objects in partially known environments by balancing exploration and exploitation. The authors propose a novel framework that integrates large language models (LLMs) with model-based planning, wherein LLM outputs are incorporated as statistical priors within model predictive control to jointly reason about semantic scene context and path costs for estimating object presence likelihood. To enhance runtime efficiency, they introduce a multi-armed bandit–based prompt selection mechanism that leverages offline replay for rapid adaptation. Simulations demonstrate that the proposed approach improves search efficiency by 11.8% over pure LLM-driven strategies and by 39.2% compared to optimistic exploration baselines. Furthermore, the prompt selection mechanism reduces average query cost by 6.5% and cumulative regret by 33.8%. Real-world robotic experiments corroborate the method’s practical effectiveness.

Technology Category

• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present a novel LLM-informed model-based planning framework, and a novel prompt selection method, for object search in partially-known environments. Our approach uses an LLM to estimate statistics about the likelihood of finding the target object when searching various locations throughout the scene that, combined with travel costs extracted from the environment map, are used to instantiate a model, thus using the LLM to inform planning and achieve effective search performance. Moreover, the abstraction upon which our approach relies is amenable to deployment-time model selection via the recent offline replay approach, an insight we leverage to enable fast prompt and LLM selection during deployment. Simulation experiments demonstrate that our LLM-informed model-based planning approach outperforms the baseline planning strategy that fully relies on LLM and optimistic strategy with as much as 11.8% and 39.2% improvements respectively, and our bandit-like selection approach enables quick selection of best prompts and LLMs resulting in 6.5% lower average cost and 33.8% lower average cumulative regret over baseline UCB bandit selection. Real-robot experiments in an apartment demonstrate similar improvements and so further validate our approach.

Problem

• Object Search
• Partially-Known Environments
• Model-based Planning
• LLM-informed Planning
• Prompt Selection

Innovation

• LLM-informed planning
• model-based planning
• prompt selection
• object search
• offline replay