DynEval: Holistic Evaluations of T2I Generative Models in the Wild

2026-07-13

AI Summary

This work addresses the challenge that existing automatic evaluation methods for text-to-image (T2I) generation models struggle to accurately identify fine-grained failure modes—such as semantic mismatches and compositional errors—at scale. To this end, the authors propose DynEval, a dynamic evaluation framework that introduces a structured pipeline to decouple assessment of text-image alignment from image quality. They further develop two novel datasets, GenDB and DynEvalInstruct, via a hierarchical prompt-and-model generation strategy. Leveraging knowledge distillation and curriculum learning, DynEval enables a compact evaluator to efficiently inherit capabilities from large models through full-parameter fine-tuning. Experiments demonstrate that DynEval significantly outperforms current evaluators across 11 benchmarks, achieves higher correlation with human judgments, and provides the first fine-grained analysis of 36 T2I models across 42 subclasses and 9 semantic dimensions.

Abstract

Recent advances in text-to-image (T2I) generation have led to models capable of producing highly realistic images. Yet, reliably evaluating their outputs remains challenging, especially at scale. Existing automatic evaluators, often relying on a static prompt set, struggle to capture subtle failure modes such as partial prompt misalignment, compositional errors, or visually plausible but semantically incorrect generations. In this work, we introduce DynEval, a Dynamic Evaluation framework designed to jointly assess text-to-image alignment and image quality of T2I models. To support scalable training beyond limited human-annotated data, we construct two large datasets. First, we build GenDB, a collection of 500K prompt-image pairs generated from human-written prompts drawn from DiffusionDB using a tiered prompt-model generation strategy. Second, building upon GenDB, we construct DynEvalInstruct, a 250K instruction dataset comprising prompt-image-response triplets distilled from a structured evaluation pipeline that decomposes evaluation into text-image alignment and visual quality reasoning. Using this dataset, we perform full fine-tuning of a compact evaluator through a curriculum learning strategy to effectively distill the superior evaluation capabilities of a larger teacher vision-language model, resulting in DynEval-2B and DynEval-4B. In extensive comparisons against existing evaluators across 11 benchmarks, our evaluator achieves a higher overall correlation with human judgments. Furthermore, it provides fine-grained analysis of the capabilities and failure modes of 36 T2I models across 42 subcategories and 9 semantic dimensions.

Problem

• text-to-image generation
• model evaluation
• alignment
• image quality
• failure modes

Innovation

• Dynamic Evaluation
• Text-to-Image Generation
• Vision-Language Model Distillation
• Curriculum Learning
• Scalable Dataset Construction