TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

2026-08-17

AI Summary

This study addresses the fragmentation and limited diagnostic utility of existing multi-reference image generation benchmarks by proposing a capability-oriented evaluation perspective. We formalize multi-reference generation as compositions of four atomic operators and construct Trace-Bench, a dataset comprising 1,600 cases. Furthermore, we introduce a novel compositional evaluation framework based on operator alignment alongside a recursive fault diagnosis tree mechanism. Experimental results precisely localize the root causes of model failures, revealing that attribute disentanglement and binding remain primary bottlenecks, with the state-of-the-art model achieving an attribute fidelity score of only 0.74. Ultimately, this work establishes an interpretable diagnostic paradigm for advancing multi-reference generation research.

Abstract

Despite recent advances in unified multimodal models for multi-reference image generation, existing benchmarks remain organized around predefined task types (e.g., "subject composition"), which are ill-suited to this combinatorial setting and lead to fragmented coverage, uncontrolled complexity, and little diagnostic value. Recognizing that diverse multi-reference tasks share a common set of atomic operations, we adopt a capability-oriented perspective and formalize four operators: Anchor ($f$), Disentangle ($g$), Apply ($\oplus$), and Compose ($C$). Any multi-reference prompt can then be represented as a compositional formula over these operators, whose structural complexity is quantified by the number of operator slots. Building on this formulation, we construct TRACE-Bench, comprising approximately 1,600 evaluation cases across slot counts 1--8, built from 631 formula templates and around 4,000 reference images spanning diverse artistic styles and real-world subjects. The formula structure directly drives an operator-aligned evaluation protocol for per-capability scoring and a diagnostic tree analysis for recursive failure localization. Evaluating 9 leading models reveals insights invisible to holistic scoring: the primary bottleneck lies in disentanglement ($g$) and attribute binding ($\oplus$) rather than scene-level composition ($C$), with even the best model scoring only 0.74 on attribute fidelity. Project page: https://amuseum-whr.github.io/TraceBench

Problem

• Multi-Reference Image Generation
• Benchmark Evaluation
• Diagnostic Analysis
• Compositional Complexity

Innovation

• Atomic Operators
• Compositional Formula
• Capability-Oriented Evaluation
• Diagnostic Tree Analysis
• TRACE-Bench