Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist

2026-06-30

AI Summary

This work addresses the limitation of existing text-to-image models in faithfully adhering to complex, multi-dimensional constraints—such as spatial relationships, artistic style, and text rendering—under intricate prompts, as current evaluation benchmarks rely predominantly on simplistic instructions. To this end, the authors introduce Arena-T2I Hard, a new benchmark comprising 310 real-world complex prompts, each decomposed into approximately 30 fine-grained binary constraints. They further propose a dependency-aware checklist reward mechanism, Group-Decoupled Preference Optimization (GDPO), and a multi-reward fusion training strategy. Evaluated on SD3.5-Medium and FLUX.1-dev, their approach significantly outperforms existing baselines, achieving state-of-the-art fidelity–aesthetics trade-offs on the MMRB2 benchmark, with the strongest closed-source system attaining a score of 0.855—33 percentage points ahead of the next-best model.

Abstract

Faithfulness -- how precisely a generated image aligns with its prompt -- is increasingly central to the real-world utility of text-to-image (T2I) models. Existing faithfulness benchmarks, however, rely on simple atomic instructions, on which top-tier systems already achieve near-perfect scores. As T2I models enter creative workflows, users issue multi-faceted requests combining intricate spatial relationships, stylistic constraints, and complex text rendering. In this setting, a single binary VLM-judge score no longer captures which specific constraints the model fails to satisfy. We introduce Arena-T2I Hard, a 310-prompt stress benchmark drawn from real arena T2I logs, with approximately 30 decomposed yes/no constraints per prompt spanning six categories, including text rendering. The strongest closed-source system we evaluate reaches 0.855 with a 33~pp performance gap across 11 systems, demonstrating substantial discriminative power. Moreover, high public-arena rankings fail to predict faithfulness, confirming that holistic Bradley-Terry (BT) preference scores prioritize aesthetics over fine-grained prompt adherence. We propose a dependency-aware checklist reward that decomposes each prompt into a DAG of yes/no questions and zeroes descendants of failed parents, turning faithfulness into a per-constraint training signal. Combined with a BT aesthetic reward via group-decoupled normalization (GDPO), which standardizes each reward within its rollout group so neither collapses, the recipe attains a strictly better faithfulness-aesthetics trade-off on SD3.5-Medium and FLUX.1-dev under MMRB2 pairwise comparisons than every single-reward, naive weighted-sum, or 4-reward BT-ensemble baseline.

Problem

• faithfulness
• text-to-image
• benchmarking
• prompt adherence
• complex constraints

Innovation

• faithfulness benchmarking
• dependency-aware checklist
• text-to-image generation
• reward decomposition
• group-decoupled normalization