Assistron: Bayesian Shared Autonomy with Off-the-shelf Vision-Language-Action Models

2026-06-22

AI Summary

This work addresses the challenges of vision-language-action (VLA) models in contact-rich everyday tasks, where they often fail and struggle to balance the burden of human-robot collaboration. The authors propose a Bayesian shared autonomy framework that operates without fine-tuning off-the-shelf VLA models. The approach leverages the VLA model to generate high-level motion trajectories and incorporates a phase-aware mechanism to trigger timely user intervention at critical failure points, with flow-matching guidance enabling real-time action adjustments. By preserving the VLA model’s general behavioral priors, the framework avoids catastrophic forgetting. Evaluated on a multi-task benchmark of everyday manipulation tasks, it achieves significantly higher success rates than fully autonomous methods while substantially reducing users’ cognitive and physical workload compared to traditional teleoperation.

Abstract

We propose Assistron, a shared autonomy model that leverages Vision-Language-Action (VLA) models to assist the user in daily activities. Our approach is grounded in two core principles: (1)~minimizing human cognitive and physical effort by leveraging VLA-driven autonomy for macro-movements, and (2)~prioritizing human intervention specifically at critical failure points. Driven by the user's verbal language commands, Assistron utilizes the VLA to autonomously execute macro-reaching trajectories, saving users' effort. In contact-rich interactions where VLAs tend to fail, Assistron employs a phase-aware interaction detection mechanism and solicits the user to intervene, in turn adjusting the VLA's action generation via flow matching guidance. Critically, our formulation eliminates the need for VLA fine-tuning, protecting its broad behavioral priors from catastrophic forgetting and ensuring the model does not become a narrow specialist. We validate our approach on a comprehensive multi-task scene recovery benchmark encompassing diverse daily manipulation skills. Empirical results demonstrate that Assistron significantly improves task success rates over pure autonomous baselines while significantly reducing human cognitive and physical workload compared to traditional teleoperation, offering a scalable, smooth, and effortless paradigm for assistive manipulation. The code is available in https://github.com/mousecpn/Assistron.git.

Problem

• shared autonomy
• Vision-Language-Action models
• human-robot interaction
• assistive manipulation
• task failure

Innovation

• Shared Autonomy
• Vision-Language-Action Models
• Flow Matching
• Phase-aware Interaction Detection
• No Fine-tuning