CombatVLA: An Efficient Vision-Language-Action Model for Combat Tasks in 3D Action Role-Playing Games

2025-03-12

AI Summary

Real-time combat decision-making in 3D action role-playing games (ARPGs) demands millisecond-level response latency, high-resolution visual perception, and dynamic tactical reasoning—posing significant challenges for existing AI agents. Method: We propose the first lightweight 3B vision-language-action (VLA) model specifically optimized for ARPG combat. Our approach introduces a novel Action-of-Thought (AoT) data format and a truncated AoT inference strategy, integrated with video-action pair training, action tracker-based data collection, and an end-to-end execution framework. Contribution/Results: Evaluated on a newly constructed ARPG combat understanding benchmark, our model achieves higher task success rates than human players while reducing inference latency to the millisecond scale and accelerating combat execution by 50×. To foster reproducibility and community advancement, we fully open-source the code, datasets, models, and benchmark.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent advances in Vision-Language-Action models (VLAs) have expanded the capabilities of embodied intelligence. However, significant challenges remain in real-time decision-making in complex 3D environments, which demand second-level responses, high-resolution perception, and tactical reasoning under dynamic conditions. To advance the field, we introduce CombatVLA, an efficient VLA model optimized for combat tasks in 3D action role-playing games(ARPGs). Specifically, our CombatVLA is a 3B model trained on video-action pairs collected by an action tracker, where the data is formatted as action-of-thought (AoT) sequences. Thereafter, CombatVLA seamlessly integrates into an action execution framework, allowing efficient inference through our truncated AoT strategy. Experimental results demonstrate that CombatVLA not only outperforms all existing models on the combat understanding benchmark but also achieves a 50-fold acceleration in game combat. Moreover, it has a higher task success rate than human players. We will open-source all resources, including the action tracker, dataset, benchmark, model weights, training code, and the implementation of the framework at https://combatvla.github.io/.

Problem

• Real-time decision-making in complex 3D environments
• High-resolution perception and tactical reasoning under dynamic conditions
• Efficient combat task execution in 3D action role-playing games

Innovation

• Efficient 3B Vision-Language-Action model for ARPGs
• Trained on action-of-thought sequences from action tracker
• Truncated AoT strategy enables 50-fold acceleration