What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities

2025-06-10

AI Summary

Existing virtual agent benchmarks suffer from uncontrolled task complexity, narrow scenario coverage, single-dimensional evaluation, and heavy reliance on manual annotation. To address these limitations, we propose OmniBench—the first self-generating, cross-platform, graph-structured, multidimensional benchmark. It leverages a subtask knowledge graph and an automated synthesis pipeline to generate controllable-complexity tasks across 20 diverse scenarios (36K tasks total). Our approach introduces a novel graph-structured task synthesis paradigm and the OmniEval framework, enabling joint assessment of subtask-level accuracy, graph-topological validity, and ten core agent capabilities. Synthesized data achieves a 91% human acceptance rate and yields higher training efficiency than manually annotated data. Comprehensive evaluation of over 20 state-of-the-art multimodal large language model (MLLM)-based agents reveals precise capability bottlenecks, facilitating quantifiable, systematic advancement of virtual agent intelligence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

As multimodal large language models (MLLMs) advance, MLLM-based virtual agents have demonstrated remarkable performance. However, existing benchmarks face significant limitations, including uncontrollable task complexity, extensive manual annotation with limited scenarios, and a lack of multidimensional evaluation. In response to these challenges, we introduce OmniBench, a self-generating, cross-platform, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity through subtask composition. To evaluate the diverse capabilities of virtual agents on the graph, we further present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities. Our synthesized dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91% human acceptance rate. Training on our graph-structured data shows that it can more efficiently guide agents compared to manually annotated data. We conduct multidimensional evaluations for various open-source and closed-source models, revealing their performance across various capabilities and paving the way for future advancements. Our project is available at https://omni-bench.github.io/.

Problem

• Limitations in existing virtual agent benchmarks
• Lack of controllable task complexity and scenarios
• Need for multidimensional evaluation framework

Innovation

• Self-generating cross-platform graph-based benchmark
• Automated pipeline for controllable complexity tasks
• Multidimensional evaluation framework OmniEval