A Survey on Quality Metrics for Text-to-Image Generation

2024-03-18

AI Summary

Text-to-image generation lacks a comprehensive evaluation framework that jointly assesses semantic alignment (prompt–image consistency) and visual fidelity. Method: We propose the first two-dimensional, structured taxonomy of quality metrics for this task, centered on “compositional quality” and “general quality.” Systematically analyzing over 30 metrics, we characterize their theoretical foundations, applicability domains, and associated benchmarks (e.g., COCO-TI, TIFA, Pick-a-Pic). Our analysis integrates literature review, taxonomic modeling, and empirical validation. Contribution/Results: We rigorously disentangle semantic alignment from visual fidelity, exposing critical limitations of existing metrics in fine-grained composition understanding, subjective perceptual modeling, and cross-domain generalization. The taxonomy fills a fundamental gap in evaluation paradigms, delivering a reproducible, extensible, and unified benchmarking framework. It provides both theoretical grounding and practical guidance for algorithm development and benchmark curation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

AI-based text-to-image models do not only excel at generating realistic images, they also give designers more and more fine-grained control over the image content. Consequently, these approaches have gathered increased attention within the computer graphics research community, which has been historically devoted towards traditional rendering techniques, that offer precise control over scene parameters (e.g., objects, materials, and lighting). While the quality of conventionally rendered images is assessed through well established image quality metrics, such as SSIM or PSNR, the unique challenges of text-to-image generation require other, dedicated quality metrics. These metrics must be able to not only measure overall image quality, but also how well images reflect given text prompts, whereby the control of scene and rendering parameters is interweaved. Within this survey, we provide a comprehensive overview of such text-to-image quality metrics, and propose a taxonomy to categorize these metrics. Our taxonomy is grounded in the assumption, that there are two main quality criteria, namely compositional quality and general quality, that contribute to the overall image quality. Besides the metrics, this survey covers dedicated text-to-image benchmark datasets, over which the metrics are frequently computed. Finally, we identify limitations and open challenges in the field of text-to-image generation, and derive guidelines for practitioners conducting text-to-image evaluation.

Problem

• Image Generation
• Evaluation Metrics
• Artificial Intelligence

Innovation

• AI Image Quality Metrics
• Text-to-Image Evaluation
• Composition and Overall Quality