Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective

2024-10-14

arXiv.org

AI Summary

Current text-to-image (T2I) models struggle to capture semantic differences induced by word-order variations, and mainstream evaluation relies on indirect metrics—e.g., text–image similarity—that fail to characterize the causal relationship between input semantic perturbations and output image generation. To address this, we propose SemVar, the first causally grounded framework for evaluating semantic variation in T2I generation. It introduces a novel metric, SemVarEffect, and a dedicated benchmark, SemVarBench, leveraging linguistically informed nontrivial word-order permutations and causal effect estimation to quantify the causal impact of input semantic changes on generated images. Experiments reveal that cross-modal alignment modules (e.g., UNet/Transformer) are more critical than text encoders; relational understanding (e.g., subject–object dependencies) is substantially weaker than attribute comprehension (0.07 vs. 0.17–0.19); and CogView-3-Plus and Ideogram 2 achieve top performance (0.2/1). Code and benchmark are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Accurate interpretation and visualization of human instructions are crucial for text-to-image (T2I) synthesis. However, current models struggle to capture semantic variations from word order changes, and existing evaluations, relying on indirect metrics like text-image similarity, fail to reliably assess these challenges. This often obscures poor performance on complex or uncommon linguistic patterns by the focus on frequent word combinations. To address these deficiencies, we propose a novel metric called SemVarEffect and a benchmark named SemVarBench, designed to evaluate the causality between semantic variations in inputs and outputs in T2I synthesis. Semantic variations are achieved through two types of linguistic permutations, while avoiding easily predictable literal variations. Experiments reveal that the CogView-3-Plus and Ideogram 2 performed the best, achieving a score of 0.2/1. Semantic variations in object relations are less understood than attributes, scoring 0.07/1 compared to 0.17-0.19/1. We found that cross-modal alignment in UNet or Transformers plays a crucial role in handling semantic variations, a factor previously overlooked by a focus on textual encoders. Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding. Our benchmark and code are available at https://github.com/zhuxiangru/SemVarBench .

Problem

• Evaluating semantic variation effects in text-to-image synthesis
• Assessing causality between input and output semantic variations
• Improving cross-modal alignment for better semantic understanding

Innovation

• Proposes SemVarEffect metric for causal evaluation
• Introduces SemVarBench benchmark for semantic variations
• Highlights cross-modal alignment in UNet or Transformers