Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs

2026-02-24

AI Summary

This work addresses the limited capacity of existing embodied large language models to reflect on task failures and learn from errors. To overcome this, we propose the first test-time planning framework that integrates human practitioner-inspired reflection mechanisms, introducing a tripartite reflection process—during-action, post-action, and retrospective—to enable effective credit assignment over long-horizon tasks. At test time, the method generates and evaluates multiple action candidates, dynamically updates its reflection model and policy using external feedback, and supports retrospective optimization for improved decision-making. Evaluated on the Long-Horizon Household and MuJoCo Cupboard Fitting benchmarks, our approach significantly outperforms baseline methods. Ablation studies confirm the complementary roles of the individual reflection mechanisms, and real-world robot experiments further demonstrate its capability to correct erroneous behaviors.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models
• Multiagent Systems: Multiagent Planning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Embodied LLMs endow robots with high-level task reasoning, but they cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials where mistakes repeat rather than accumulate into experience. Drawing upon human reflective practitioners, we introduce Reflective Test-Time Planning, which integrates two modes of reflection: \textit{reflection-in-action}, where the agent uses test-time scaling to generate and score multiple candidate actions using internal reflections before execution; and \textit{reflection-on-action}, which uses test-time training to update both its internal reflection model and its action policy based on external reflections after execution. We also include retrospective reflection, allowing the agent to re-evaluate earlier decisions and perform model updates with hindsight for proper long-horizon credit assignment. Experiments on our newly-designed Long-Horizon Household benchmark and MuJoCo Cupboard Fitting benchmark show significant gains over baseline models, with ablative studies validating the complementary roles of reflection-in-action and reflection-on-action. Qualitative analyses, including real-robot trials, highlight behavioral correction through reflection.

Problem

• Embodied LLMs
• reflection
• trial-and-error learning
• long-horizon tasks
• credit assignment

Innovation

• Reflective Test-Time Planning
• Embodied LLMs
• reflection-in-action
• reflection-on-action
• test-time training