Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition

2026-07-07

AI Summary

This work addresses execution failures in long-horizon household tasks caused by semantic handoff errors between skills—where a preceding skill succeeds but fails to establish a reliable initial state for the next. To tackle this, the authors propose an agent coordination framework built upon BEHAVIOR-1K, integrating a multi-view vision-language model to validate task states at skill boundaries. The approach combines π₀.₅ checkpoints, typed parameters, and step budgets to enable targeted retries or replanning. For the first time, the study systematically distinguishes single-skill performance from compositional robustness, uncovering a representational gap between clean demonstrations and chained execution, thereby transforming near-zero end-to-end success rates into diagnosable failure modes. Experiments reveal that while navigation, grasping, placing, and door-opening skills achieve 77–100% success in isolation, their performance degrades significantly in chained settings, exposing critical bottlenecks in readiness assessment, target localization, and low-level control.

Abstract

Long-horizon household tasks require robots to compose many language-conditioned skills, yet the boundary between consecutive skills is rarely explicit. A skill may satisfy its own postcondition while leaving the robot, objects, or camera views in a state from which the next skill cannot reliably start. We study this semantic handoff problem in BEHAVIOR-1K through an agent-orchestrated vision-language-action execution harness. The harness invokes $π_{0.5}$-based skill checkpoints trained from cleaned BEHAVIOR-1K demonstrations, assigns each skill typed arguments and a step budget, and uses multi-view vision-language model verification to decide whether execution should advance, retry, or replan. To separate isolated skill competence from long-horizon compositional robustness, we evaluate the same checkpoints under two initial-state distributions: clean skill-boundary snapshots and chained terminal states produced by previous skills. Selected navigation, grasping, placement, and door-opening skills achieve 77--100% success from clean snapshots under human-reviewed verification, yet composed rollouts still frequently stall from chained states. Execution traces attribute these failures to next-skill readiness, target grounding, and low-level control execution, revealing a substantial gap between single-skill success and reliable long-horizon task completion. These findings turn near-zero end-to-end task success into actionable diagnostics, showing that future VLA skill libraries must learn robustness to the messy chained-state distribution that clean demonstrations systematically underrepresent.

Problem

• semantic handoff
• skill composition
• vision-language-action
• long-horizon tasks
• chained states

Innovation

• semantic handoff
• vision-language-action
• skill composition
• multi-view verification
• chained-state robustness