FAGER: Factually Grounded Evaluation and Refinement of Text-to-Image Models

2026-05-18

AI Summary

This work addresses the critical gap in existing text-to-image generation models, which often lack effective mechanisms to ensure factual correctness—particularly for scientific, historical, product-related, or cultural content—and struggle with facts that are implicit in prompts or require external knowledge. To tackle this, the authors propose FAGER, a novel framework that leverages large language models to extract structured facts and generate corresponding question-answer pairs. FAGER integrates reference image–guided visual verification with vision-language models to enable fine-grained, interpretable assessment of factual consistency. Notably, it establishes the first training-free closed-loop generation-feedback optimization pipeline for enhancing factuality. Experiments across five benchmark datasets demonstrate that FAGER significantly outperforms existing metrics in factual A/B testing and effectively improves the factual accuracy of generated images.

Abstract

Existing text-to-image (T2I) evaluation metrics mainly assess whether generated images align with information explicitly stated in the prompt, but often fail to capture factual requirements that are implicit, externally grounded, or identity-defining. As a result, they are not well suited for evaluating factual correctness in prompts involving scientific knowledge, historical facts, products, or culture-specific concepts. We propose FActually Grounded Evaluation and Refinement (FAGER), an agentic framework that evaluates whether generated images correctly reflect visually verifiable facts grounded in or implied by the prompt, while also providing actionable feedback for improvement. FAGER first constructs a structured factual rubric by combining LLM-based fact proposal with reference-guided visual fact extraction and verification, then converts the rubric into question-answer pairs for VLM-based evaluation. To validate FAGER as a factuality metric, we introduce a Factual A/B test, which measures whether a metric prefers factual reference images over corresponding generated images. Across five datasets spanning science, history, products, culture, and knowledge-intensive concepts, FAGER consistently outperforms prior metrics on this test. We further show that FAGER can be used to refine T2I outputs in a fully training-free manner, yielding substantial factuality gains across datasets.

Problem

• text-to-image evaluation
• factuality
• implicit facts
• visually verifiable facts
• grounded knowledge

Innovation

• factuality evaluation
• text-to-image generation
• visual grounding
• LLM-VLM collaboration
• training-free refinement