Continually Evolving Skill Knowledge in Vision Language Action Model

2025-11-22

AI Summary

Existing Vision-Language-Action (VLA) models lack continual skill learning capability, while mainstream continual learning approaches suffer from poor scalability and high computational overhead. To address these limitations, this paper proposes Stellar VLA—a novel framework that jointly models task representations and a hierarchical skill knowledge space. Stellar VLA achieves task specialization and efficient knowledge retention without incremental parameters, via co-learning of task embeddings and knowledge space, knowledge-guided expert routing, and self-supervised knowledge evolution. Its TS-Stellar variant further enhances complex action reasoning. Evaluated on the LIBERO benchmark and real-world robotic tasks, Stellar VLA achieves an average success rate improvement of over 50 percentage points compared to state-of-the-art baselines. These results demonstrate the effectiveness and scalability of knowledge-driven continual learning for general-purpose robotic intelligence.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Developing general robot intelligence in open environments requires continual skill learning. Recent Vision-Language-Action (VLA) models leverage massive pretraining data to support diverse manipulation tasks, but they still depend heavily on task-specific fine-tuning, revealing a lack of continual learning capability. Existing continual learning methods are also resource-intensive to scale to VLA models. We propose Stellar VLA, a knowledge-driven continual learning framework with two variants: T-Stellar, modeling task-centric knowledge space, and TS-Stellar, capturing hierarchical task-skill structure. Stellar VLA enables self-supervised knowledge evolution through joint learning of task latent representation and the knowledge space, reducing annotation needs. Knowledge-guided expert routing provide task specialization without extra network parameters, lowering training overhead.Experiments on the LIBERO benchmark and real-world tasks show over 50 percentage average improvement in final success rates relative to baselines. TS-Stellar further excels in complex action inference, and in-depth analyses verify effective knowledge retention and discovery. Our code will be released soon.

Problem

• Developing general robot intelligence requires continual skill learning in open environments
• Existing Vision-Language-Action models lack continual learning capability and depend on fine-tuning
• Current continual learning methods are resource-intensive and difficult to scale to VLA models

Innovation

• Knowledge-driven continual learning framework for VLA
• Self-supervised knowledge evolution reduces annotation needs
• Knowledge-guided expert routing lowers training overhead