Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memory

2026-08-17

AI Summary

This study addresses error accumulation and subtask transition failures in long-horizon robotic manipulation by proposing the Baton framework. By treating subtasks as exploration units, the method reduces search complexity from multiplicative to additive and introduces a transition-aware memory mechanism to effectively resolve state handover and entry condition matching challenges. Leveraging frozen Vision-Language-Action models alongside LLM-based agent planning, Baton enables efficient composition and robust execution without parameter updates. Evaluated on the RoboMemArena benchmark, the framework achieves an 11.6% improvement in task success rate and a 14.9% increase in cumulative success rate, significantly outperforming existing state-of-the-art methods.

Abstract

Long-horizon robot manipulation chains many contact-rich skills into one multi-stage task. Vision-language-action (VLA) models increasingly master the individual skills, yet the chain still fails: errors compound beyond the policy's ability to correct, and one subtask silently constrains the next. A promising recipe freezes the VLA and puts an LLM agent in charge: it plans in language, moves in free space with analytic primitives, invokes the VLA only for contact-rich segments, and writes adaptation into language memory. Applied to long horizons, it breaks twice. (1) Competence comes from whole-task exploration at test time, whose cost is multiplicative in stages: if one stage needs T episodes, a K-stage task needs about T^K, and a failure does not reveal which stage caused it. (2) It has no representation of transitions: the VLA primitive carries an exit but no entry condition, so a subtask can succeed in a form its successor cannot use. We present BATON. Against (1), BATON makes the subtask the unit of exploration: each is explored in the cheap short-horizon regime and its solution stored in memory; a long-horizon trajectory is then composed from these solutions rather than discovered whole. Cost becomes additive (T*K) and every failure is attributed to a single stage. Against (2), BATON equips exploration with a transition-aware memory. Within a subtask, a verifier agent governs the invocation transition: the VLA is called only after the wrist view confirms the scene is ready. Across subtasks, a handoff transition restores an entry state disturbed by the predecessor's residue, and a lookahead transition selects the strategy whose outcome the successor can inherit. No parameters are updated. On the long-horizon benchmark RoboMemArena, BATON improves task success by 11.6% and cumulative success by 14.9% over the SoTA.

Problem

• Long-horizon robot manipulation
• Subtask exploration
• Transition-aware memory
• Vision-language-action models
• Error compounding

Innovation

• Agentic Subtask Exploration
• Transition-aware Memory
• Long-Horizon Manipulation
• Vision-Language-Action Models
• Test-time Adaptation