Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

2026-06-25

AI Summary

Current evaluations of unified multimodal models often treat understanding and generation capabilities in isolation, neglecting their synergistic interplay. This work introduces Unison, a benchmark comprising 2,169 high-quality task samples, which for the first time systematically assesses model synergy along three dimensions: internal consistency, mutual guidance, and reciprocal enhancement. To enable fine-grained analysis, the authors design both unified and decoupled diagnostic pathways and develop Unison-Judge, an automatic scoring model aligned with human preferences. Their findings uncover critical limitations in existing models’ ability to jointly perform understanding and generation, offering clear directions for future research. The dataset and evaluation toolkit are publicly released to support further advancements in multimodal foundation models.

Abstract

Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human Alignment: we also introduce Unison-Judge, an evaluation model well aligned with human judgments to ensure reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified multimodal systems and highlight promising directions for future research. Codes, Unison and Unison-Judge are publicly available at https://github.com/FudanCVL/Unison.

Problem

• unified multimodal models
• synergistic understanding and generation
• benchmarking
• comprehensive evaluation
• human alignment

Innovation

• unified multimodal models
• synergistic evaluation
• understanding-generation synergy
• diagnostic benchmarking
• human-aligned evaluation