Bridge Thinking and Acting: Unleashing Physical Potential of VLM with Generalizable Action Expert

2025-10-04

AI Summary

Current vision-language-action (VLA) models exhibit poor generalization: single-system approaches are constrained by narrow-domain training data, while dual-system architectures suffer from semantic ambiguity in the action module, hindering cross-task training and deployment. Method: We propose a novel decoupled “thinking-and-acting” paradigm, introducing the first generalizable Action Expert framework. It leverages sparse 3D waypoints as a semantic bridge between high-level planning and low-level control—where a vision-language model (VLM) generates coarse waypoints, and the Action Expert synthesizes dense, executable actions from real-time point clouds. We establish a new “action pretraining–point-cloud fine-tuning” pipeline and formally define the collaboration protocol between the two subsystems. Contribution/Results: Experiments demonstrate zero-shot transfer to unseen tasks and environments, significantly improving cross-domain generalization and physical interaction efficiency.

Technology Category

• Computer Vision: Language and Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Although Vision-Language Models (VLM) have demonstrated impressive planning and reasoning capabilities, translating these abilities into the physical world introduces significant challenges. Conventional Vision-Language-Action (VLA) models, which integrate reasoning and action into a monolithic architecture, generalize poorly because they are constrained by scarce, narrow-domain data. While recent dual-system approaches attempt to decouple "thinking" from "acting", they are often constrained by semantic ambiguities within the action module. This ambiguity makes large-scale, cross-task training infeasible. Consequently, these systems typically necessitate fine-tuning on newly collected data when deployed to novel environments, and the cooperation mechanism between the two systems remains ill-defined. To address these limitations, we introduce, for the first time, a framework centered around a generalizable action expert. Our approach utilizes sparse 3D trajectories as an intermediate representation, effectively bridging the high-level planning capabilities of the VLM with the low-level physical action module. During the planning phase, the VLM is only required to generate coarse 3D waypoints. These waypoints are then processed by our generalizable action expert, which refines them into dense, executable action sequences by sampling real-time point cloud observations of the environment. To promote training efficiency and robust generalization, we introduce a novel "Action Pre-training, Pointcloud Fine-tuning" paradigm. Our method combines the broad generalization capabilities of VLMs in visual understanding and planning with the fine-grained, action-level generalization of action expert.

Problem

• VLM models struggle to translate reasoning into physical actions effectively
• Current VLA systems generalize poorly due to semantic ambiguities in action modules
• Existing approaches require fine-tuning for new environments and lack clear cooperation mechanisms

Innovation

• Generalizable action expert bridges VLM planning and physical actions
• Sparse 3D trajectories serve as intermediate representation between systems
• Action pre-training with pointcloud fine-tuning enables robust generalization