A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulation

2024-05-27

AI Summary

Existing vision-language-action (VLA) models exhibit limited robustness on novel, complex tasks and lack human-like reasoning and error-correction capabilities. To address this, we propose a dual-system VLA framework endowed with self-correcting ability: a “fast system” for intuitive, real-time action prediction, and a “slow system” for deliberate failure attribution and iterative correction. Our contributions include: (1) the first fast-slow collaborative architecture for VLAs; (2) a chain-of-thought–driven failure attribution mechanism coupled with expert-feedback-guided progressive correction; and (3) continual policy learning from successfully corrected trajectories. Technically, the framework integrates parameter-efficient fine-tuning, SE(3) pose prediction, multimodal large language models (MLLMs), and continuous policy optimization. Experiments in both simulation and real-robot settings demonstrate substantial improvements in manipulation accuracy and generalization—particularly on unseen tasks—where our method achieves superior error-correction efficiency over state-of-the-art approaches.

Technology Category

• Computer Vision: Large Vision Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Manipulation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recently, some studies have integrated Multimodal Large Language Models into robotic manipulation, constructing vision-language-action models (VLAs) to interpret multimodal information and predict SE(3) poses. While VLAs have shown promising progress, they may suffer from failures when faced with novel and complex tasks. To emulate human-like reasoning for more robust manipulation, we propose the self-corrected (SC-)VLA framework, which integrates fast system for directly predicting actions and slow system for reflecting on failed actions within a single VLA policy. For the fast system, we incorporate parameter-efficient fine-tuning to equip the model with pose prediction capabilities while preserving the inherent reasoning abilities of MLLMs. For the slow system, we propose a Chain-of-Thought training strategy for failure correction, designed to mimic human reflection after a manipulation failure. Specifically, our model learns to identify the causes of action failures, adaptively seek expert feedback, reflect on the current failure scenario, and iteratively generate corrective actions, step by step. Furthermore, a continuous policy learning method is designed based on successfully corrected samples, enhancing the fast system's adaptability to the current configuration. We compare SC-VLA with the previous SOTA VLA in both simulation and real-world tasks, demonstrating an efficient correction process and improved manipulation accuracy on both seen and unseen tasks.

Problem

• Enhances robotic manipulation with self-correcting visionlanguageaction model.
• Integrates fast and slow systems for robust task handling.
• Improves accuracy and adaptability in novel and complex tasks.

Innovation

• Self-corrected VLA framework for robust manipulation
• Parameter-efficient fine-tuning for pose prediction
• Chain-of-Thought training for failure correction