AgentStudio: A Toolkit for Building General Virtual Agents

2024-03-26

arXiv.org

AI Summary

Existing virtual agent development environments are often domain-specific and suffer from complex configurations, hindering realistic scenario evaluation and lacking fine-grained assessment of foundational capabilities. Method: We propose a lightweight, open-domain interactive environment supporting multimodal video observation and GUI/API-based input/output, integrated with task construction, GUI/video action annotation, and other tooling. We introduce the first “Environment–Tool–Benchmark” tripartite framework. Contribution/Results: We release three decoupled, complementary benchmarks—GroundUI (GUI grounding), IDMBench (video understanding and manipulation), and CriticBench (success detection and causal attribution)—addressing critical evaluation gaps. Furthermore, we establish the first open-source agent development kit enabling end-to-end training, automated evaluation, and semi-automated annotation, significantly improving measurability of agent generalization and self-improvement capabilities in open-world settings.

Technology Category

• Multiagent Systems: Adversarial Agents
• Computer Vision: Video Understanding & Activity Analysis
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

General virtual agents need to handle multimodal observations, master complex action spaces, and self-improve in dynamic, open-domain environments. However, existing environments are often domain-specific and require complex setups, which limits agent development and evaluation in real-world settings. As a result, current evaluations lack in-depth analyses that decompose fundamental agent capabilities. We introduce AgentStudio, a trinity of environments, tools, and benchmarks to address these issues. AgentStudio provides a lightweight, interactive environment with highly generic observation and action spaces, e.g., video observations and GUI/API actions. It integrates tools for creating online benchmark tasks, annotating GUI elements, and labeling actions in videos. Based on our environment and tools, we curate an online task suite that benchmarks both GUI interactions and function calling with efficient auto-evaluation. We also reorganize existing datasets and collect new ones using our tools to establish three datasets: GroundUI, IDMBench, and CriticBench. These datasets evaluate fundamental agent abilities, including GUI grounding, learning from videos, and success detection, pointing to the desiderata for robust, general, and open-ended virtual agents.

Problem

• Developing general virtual agents
• Overcoming domain-specific limitations
• Evaluating fundamental agent capabilities

Innovation

• Multimodal observation handling
• Generic action space integration
• Auto-evaluation benchmark tasks