GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning

2026-02-04

AI Summary

This work addresses the limited zero-shot generalization capability of existing vision-language-action models in open-world robotic manipulation. The authors propose a hierarchical vision-language-action framework that operates without real robot data or human demonstrations. At the high level, a fine-tuned Affordance Segmentation Module identifies actionable keypoints; at the mid level, a 3DAgent integrates task semantics and skill priors to generate 3D manipulation trajectories; and at the low level, precise control is executed. This approach achieves, for the first time, fully zero-shot robotic manipulation alongside automatic generation of high-quality demonstration data. Evaluated on 14 tasks, it significantly outperforms state-of-the-art methods such as VoxPoser. Behavior cloning policies trained on the generated data further demonstrate enhanced robustness and scalability.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Language and Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Large foundation models have shown strong open-world generalization to complex problems in vision and language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is that the models exhibit limited zero-shot capability, which hampers their ability to generalize effectively to unseen scenarios. In this work, we propose GeneralVLA (Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning), a hierarchical vision-language-action (VLA) model that can be more effective in utilizing the generalization of foundation models, enabling zero-shot manipulation and automatically generating data for robotics. In particular, we study a class of hierarchical VLA model where the high-level ASM (Affordance Segmentation Module) is finetuned to perceive image keypoint affordances of the scene; the mid-level 3DAgent carries out task understanding, skill knowledge, and trajectory planning to produce a 3D path indicating the desired robot end-effector trajectory. The intermediate 3D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Compared to alternative approaches, our method requires no real-world robotic data collection or human demonstration, making it much more scalable to diverse tasks and viewpoints. Empirically, GeneralVLA successfully generates trajectories for 14 tasks, significantly outperforming state-of-the-art methods such as VoxPoser. The generated demonstrations can train more robust behavior cloning policies than training with human demonstrations or from data generated by VoxPoser, Scaling-up, and Code-As-Policies. We believe GeneralVLA can be the scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting. Code: https://github.com/AIGeeksGroup/GeneralVLA. Website: https://aigeeksgroup.github.io/GeneralVLA.

Problem

• zero-shot generalization
• vision-language-action models
• robotic manipulation
• trajectory planning
• foundation models

Innovation

• Vision-Language-Action
• Zero-shot Manipulation
• Knowledge-Guided Planning
• Trajectory Generation
• Foundation Models