InSight: Self-Guided Skill Acquisition via Steerable VLAs

2026-06-23

AI Summary

This work addresses the limitation of existing vision-language-action (VLA) models, which rely on static training data and struggle to autonomously acquire new skills. The authors propose InSight, a framework that for the first time enables VLA models to be steerable at the raw action level. By leveraging a vision-language model to drive task decomposition, end-effector pose analysis, and automated demonstration segmentation, the system can identify missing skills and autonomously generate, annotate, and integrate new demonstration data, establishing a self-sustaining data flywheel without human intervention. Evaluated in both simulation and real-world settings, InSight successfully acquires fundamental skills—such as block flipping, drawer closing, sweeping, twisting, and pouring—and composes them to execute novel long-horizon tasks, significantly enhancing the autonomous continual learning capability of VLA systems.

Abstract

Vision-language-action (VLA) models can learn manipulation skills from demonstrations, but their capabilities are bounded by the skills in the training data. We present InSight, a framework that unlocks autonomous skill acquisition by rendering VLAs steerable at the primitive-action level (e.g., "move gripper to the bowl", "lift upward", "pour the bottle"). InSight consists of two primary stages: (1) an automated segmentation pipeline that partitions demonstrations into labeled primitives via VLM plan decomposition and end-effector poses to enable VLA primitive steerability, and (2) a VLM-guided data flywheel that identifies missing primitives required to accomplish a novel task, autonomously attempts demonstrations of the missing primitives with VLM-proposed low-level control, and automatically labels, stores, and integrates successful demonstrations into the VLA training set. We evaluate InSight across simulation and real-world manipulation tasks, including block flipping, drawer closing, sweeping, twisting, and pouring, without any human demonstrations of these target skills. Once learned, these primitives can be composed to execute novel, long-horizon tasks without additional human demonstrations. Our findings demonstrate that primitive steerability provides a practical foundation for continual skill acquisition in VLA policies. Project website: https://insight-vla.github.io.

Problem

• vision-language-action
• skill acquisition
• primitive actions
• autonomous learning
• manipulation tasks

Innovation

• steerable VLA
• primitive-action decomposition
• autonomous skill acquisition
• VLM-guided data flywheel
• continual learning