ColorConceptBench: A Benchmark for Probabilistic Color-Concept Understanding in Text-to-Image Models

2026-01-23

AI Summary

This work addresses the limited ability of current text-to-image (T2I) models to accurately interpret implicit color concepts and their insensitivity to abstract semantic cues. To this end, we introduce ColorConceptBench, the first human-annotated benchmark comprising 1,281 implicit color concepts, and propose a novel probabilistic color distribution modeling approach that moves beyond conventional evaluation paradigms reliant on explicit color names or codes. Leveraging 6,369 high-quality human annotations, we systematically evaluate seven state-of-the-art T2I models and reveal a consistent deficiency in understanding abstract color semantics—a limitation that persists despite increases in model scale or the use of standard guidance strategies. Our study establishes a new evaluation framework and provides empirical evidence for advancing color-aware semantic comprehension in T2I generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

While text-to-image (T2I) models have advanced considerably, their capability to associate colors with implicit concepts remains underexplored. To address the gap, we introduce ColorConceptBench, a new human-annotated benchmark to systematically evaluate color-concept associations through the lens of probabilistic color distributions. ColorConceptBench moves beyond explicit color names or codes by probing how models translate 1,281 implicit color concepts using a foundation of 6,369 human annotations. Our evaluation of seven leading T2I models reveals that current models lack sensitivity to abstract semantics, and crucially, this limitation appears resistant to standard interventions (e.g., scaling and guidance). This demonstrates that achieving human-like color semantics requires more than larger models, but demands a fundamental shift in how models learn and represent implicit meaning.

Problem

• text-to-image models
• color-concept association
• implicit semantics
• probabilistic color understanding
• color semantics

Innovation

• probabilistic color understanding
• implicit color concepts
• text-to-image models
• color-concept association
• human-annotated benchmark