Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models

2026-08-12

AI Summary

Existing evaluation methods treat visual generation and understanding as disjoint capabilities, failing to holistically assess the system-level performance of unified multimodal models. This work proposes a Self-Generated Understanding (SGU) framework, introducing a novel annotation-free semantic closed-loop evaluation paradigm: a model first describes an image, then reconstructs visual content from its own generated text, and finally performs zero-shot reasoning on the reconstructed output. SGU uniquely integrates generation and understanding into a single assessment pipeline, leveraging a self-feedback mechanism to uncover latent deficiencies that manifest specifically within the model’s self-generated context. Experiments reveal that even high-performing models exhibit substantially degraded reasoning capabilities under SGU, exposing systemic limitations invisible to conventional isolated evaluations.

Abstract

As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge. Current evaluation protocols predominantly treat generative and discriminative capabilities as separate tasks, leaving a gap in system-level evaluation for unified multimodal models (UMMs). In this work, we propose Self-Generative-Understanding (SGU), a novel, annotation-free evaluation framework that probes the integrated capabilities of unified models through a semantic closed-loop challenge. Without requiring new annotations, SGU leverages the dual understanding-and-generation abilities of UMMs by asking them to first perceive an image and produce a textual description, subsequently reconstruct a visual context based on that description, and finally perform reasoning over the self-generated output. This pipeline provides a zero-cost testbed that yields an integrated performance score specifically tailored for evaluating UMMs as unified systems. Extensive experiments show that even high-performing UMMs often struggle to reason over their own generated contexts, revealing limitations that are not captured by separate evaluations of understanding or generation alone. Our work provides a complementary holistic evaluation framework and offers a foundation for benchmarking the development of next-generation unified multimodal models.

Problem

• unified multimodal models
• holistic evaluation
• vision-language models
• semantic closed-loop
• integrated capabilities

Innovation

• Self-Generative-Understanding
• semantic closed-loop
• unified multimodal models
• holistic evaluation
• annotation-free