OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

2026-07-16

AI Summary

Existing AI agent evaluation benchmarks suffer from narrow, single-scenario designs that hinder systematic assessment of general-purpose agents across heterogeneous environments. This work proposes the first multi-scenario benchmark tailored for general AI agents, featuring a three-tier domain taxonomy spanning ToC, ToB, and ToE contexts, a ten-dimensional capability framework, and eight atomic difficulty factors to enable fine-grained, diagnostic evaluation. The benchmark synthesizes executable tasks—supporting both single- and multi-turn interactions—via four task-generation pathways: DAG, DAG-S, Solver, and Program, while modeling a structured state space. The released benchmark comprises 1,431 tasks, including a high-difficulty subset of 644 tasks; state-of-the-art models such as Claude-Sonnet-5 and GPT-5.6-Sol achieve Pass@1 scores of only 58.54 and 57.14, respectively, revealing substantial deficiencies in planning and constraint maintenance.

Abstract

Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction. However, existing agent benchmarks often focus on limited scenarios, tool ecosystems, or interaction formats, making it difficult to systematically characterize model capabilities across heterogeneous application settings. We introduce OmniaBench, a benchmark for evaluating general agents across diverse scenarios with explicit state spaces. We derive application-oriented scenario knowledge from app stores, product documents, industry resources, Web retrieval, and human refinement, forming a hierarchical taxonomy that spans ToC, ToB and ToE with 90 level-1 and 354 level-2 domains. Based on this taxonomy, we construct executable environments and synthesize single-turn and multi-turn tasks through four complementary routes: DAG, DAG-S, Solver, and Program. OmniaBench further introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors to support fine-grained evaluation and analysis. The resulting dataset contains 1,431 tasks, together with a challenging subset of 644 tasks designed to reduce evaluation cost and mitigate potential contamination of the full set after public release. The bench presents substantial challenges to current frontier models, with even Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively. Further analyses reveal clear differences across domains and capabilities, as well as persistent limitations in planning, constraint maintenance, and adaptive correction. OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents.

Problem

• AI agent benchmarking
• general AI agents
• heterogeneous scenarios
• capability evaluation
• task diversity

Innovation

• general AI agents
• hierarchical taxonomy
• executable environments
• capability evaluation
• atomic difficulty factors