Imagining Recovery: Inference-Time Counterfactual Realignment for Vision-Language-Action Models

2026-08-14

AI Summary

This study addresses the limitation of Vision-Language-Action models that rely on failure data and retraining for online recovery. To overcome this, we propose CoRe, a training-free framework that synthesizes counterfactual trajectories through observational imagination during inference and minimizes robot-scene alignment errors. This approach enables unified handling of instruction changes and physical perturbations without trial-and-error. Experimental results demonstrate that CoRe improves task success rates by up to 85.0 percentage points, restoring performance to near-normal levels while reducing physical recovery interventions by 42.2%. Consequently, this work achieves efficient online adaptive recovery for frozen models, eliminating the need for costly retraining or additional failure data collection.

Abstract

Vision-language-action (VLA) models have improved the flexibility and generality of robotic manipulation, yet they remain fragile to online disruptions, such as changes in task goal, scene configuration, or robot state. Existing recovery methods often require failure data, policy retraining, or external corrective agents, introducing additional data requirements and execution risks. We propose Counterfactual Realignment (CoRe), a training-free framework that recovers a frozen VLA at inference time without failure data. Upon detecting a deviation, CoRe imagines how the policy would continue toward the current goal from a recent viable state, using synthesized observations in place of physical execution, and then minimally realigns the robot and scene to rejoin this imagined continuation before returning control to the policy. Recovery is therefore planned without physical trial-and-error, preserves completed task progress, and handles both mid-episode instruction changes and physical perturbations in a unified manner. Extensive experiments across multiple simulators, VLA backbones, and real-world settings show that CoRe improves success rates by up to 85.0 percentage points to near-nominal levels while reducing physical restorations by 42.2%, without policy fine-tuning or failure-specific recovery training.

Problem

• Vision-Language-Action Models
• Online Disruptions
• Robotic Manipulation Recovery
• Training-free Recovery

Innovation

• Counterfactual Realignment
• Training-free Recovery
• Inference-Time Adaptation
• Vision-Language-Action Models
• Synthesized Observations