Orchestra-o1: Omnimodal Agent Orchestration

2026-06-10

AI Summary

Existing agent orchestration frameworks struggle to effectively handle complex scenarios involving coexisting multimodal data such as text, images, audio, and video. To address this challenge, this work proposes Orchestra-o1-8B, a unified multimodal agent orchestration framework that enables comprehensive understanding and coordination across all modalities. Its core innovations include modality-aware task decomposition, online sub-agent specialization, parallel subtask execution, and a decision-aligned group relative policy optimization (DA-GRPO) algorithm. These components collectively enhance the generalization capability and collaborative efficiency of multimodal agents. Evaluated on the OmniGAIA benchmark, the proposed method achieves a 10.3% absolute improvement in accuracy over the previous best approach, establishing state-of-the-art performance among open-source fully multimodal agent systems.

Abstract

The recent success of agent swarms has shifted the paradigm of large language model (LLM)-based agents from single-agent workflows to multi-agent systems, highlighting the importance of agent orchestration for task decomposition and collaboration. However, existing orchestration frameworks are limited to a narrow set of modalities and struggle to generalize to more complex settings where heterogeneous modalities coexist and interact. This limitation becomes particularly pronounced in omnimodal scenarios, where tasks require the unified understanding and coordination of diverse inputs such as text, image, audio, and video. In this work, we propose Orchestra-o1, an omnimodal agent orchestration framework designed to support efficient agent collaboration across multiple modalities. Orchestra-o1 introduces a unified orchestration mechanism that enables modality-aware task decomposition, online sub-agent specialization, and parallel sub-task execution. This scalable design allows agent systems to effectively tackle complex real-world tasks involving heterogeneous information sources, surpassing the second-best approach by 10.3% accuracy on the OmniGAIA benchmark. Furthermore, we introduce decision-aligned group relative policy optimization (DA-GRPO), an efficient agentic reinforcement learning approach for training Orchestra-o1-8B, which also achieves state-of-the-art performance against all existing open-source omnimodal agents.

Problem

• omnimodal
• agent orchestration
• heterogeneous modalities
• task decomposition
• multi-agent collaboration

Innovation

• omnimodal agent orchestration
• modality-aware task decomposition
• online sub-agent specialization
• parallel sub-task execution
• decision-aligned group relative policy optimization