Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation

2026-05-27

AI Summary

This work proposes the first creator-centered, fine-grained evaluation framework for text-to-image (T2I) generation, addressing the limitations of existing benchmarks that primarily assess basic alignment while neglecting the realism and creative expressiveness required in authentic artistic practice. Integrating expert insights from professional artists, the framework establishes a hierarchical structure of 56 verifiable metrics and conducts comprehensive multidimensional evaluation using 1,000 stratified prompts. For the first time, professional art creation workflows are incorporated into T2I assessment through a unified judgment model, Q-Judger (based on Qwen3.6-27B), trained under the supervision of 80 experts from global art institutions and enhanced with blind labeling and triple-review mechanisms. Experiments demonstrate that the framework effectively discriminates between state-of-the-art models and significantly outperforms existing benchmarks in evaluating realism and creative generation, offering reliable and interpretable signals for industrial-scale model optimization.

Abstract

Text-to-Image generation has evolved from basic image synthesis into a frequently used core capability in professional creative workflows, where simple text-image alignment can no longer satisfy users' pressing demands for faithful real-world reconstruction and genuine creative expression. Existing benchmarks, however, remain anchored in these foundational criteria and do not yet capture the nuanced capabilities that matter in authentic artistic practice, making it difficult to reliably distinguish state-of-the-art T2I models. To address the gap, we introduce Qwen-Image-Bench, a creator-centric benchmark co-designed with professional artists and grounded in real-world creation scenarios. Qwen-Image-Bench enriches conventional evaluation with two application-driven dimensions: Real-world Fidelity and Creative Generation. Drawing on the staged reasoning inherent in professional artistic workflows, we organize these five pillars into a top-down hierarchical taxonomy that further decomposes into 23 second-level sub-capabilities and 56 third-level verifiable rubrics. To ensure broad coverage, we curate 1000 stratified prompts with each prompt jointly exercising more than four fine-grained facets across multiple pillars. We train a unified judge model Q-Judger based on Qwen3.6-27B, supervised by 80 professional annotators from global art academies under blind labeling and triple-review protocols, that scores every image across all 56 verifiable facets, producing fine-grained, rubric-grounded, and fully attributable diagnostics rather than a single opaque score. Empirically, Qwen-Image-Bench reliably distinguishes leading T2I models, achieving the greatest separation on the two application-driven dimensions of Real-world Fidelity and Creative Generation where existing benchmarks provide little insight, while also providing a trustworthy optimization signal for production-level T2I development.

Problem

• Text-to-Image Evaluation
• Real-world Fidelity
• Creative Generation
• Benchmarking
• Artistic Workflows

Innovation

• text-to-image evaluation
• creator-centric benchmark
• real-world fidelity
• creative generation
• hierarchical rubric