Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

2026-07-09

AI Summary

Existing vision-language-action (VLA) models exhibit limited generalization under perturbations such as semantic re-targeting, object re-binding, layout changes, and unstable contacts, while purely analytical primitives struggle with irregular grasps and complex interactions. This work proposes a memory-augmented agent framework that decouples a frozen VLA—used as a retry-capable primitive for contact-intensive manipulation—from a small set of fixed analytical primitives. A memory-guided planner handles non-contact phases and semantic re-localization, invoking the VLA only during localized contact stages. The system learns the applicability boundaries of each primitive through execution trajectories, success heuristics, and failure models, thereby extending the VLA’s capabilities without fine-tuning. The approach outperforms the strongest baseline by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and achieves a 58.4% success rate on RoboTwin C2R.

Abstract

Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R.

Problem

• Vision-Language-Action
• manipulation primitives
• distribution shift
• contact-rich control
• semantic retargeting

Innovation

• Vision-Language-Action
• memory-augmented agent
• manipulation primitives
• zero-shot generalization
• contact-rich control