Beyond Color Geometry: Evaluating Human-Like Color Representations in Vision Models

2026-07-15

AI Summary

Current evaluations of color representation in vision models predominantly rely on geometric spaces or discrete labels, which fail to capture the fuzzy and graded nature of human color perception. This work proposes a perceptual evaluation framework grounded in 86 fuzzy color categories, systematically assessing vision encoders along three dimensions: category boundary fidelity, representational compactness, and hierarchical alignment beyond geometric structure. The framework introduces, for the first time, a fuzzy color system calibrated to human survey data and integrates it with feature analysis and comparative experiments on Vision Transformer (ViT) architectures. Among eleven ViT encoders examined, masked autoencoders (MAE) demonstrate superior non-geometric alignment, producing global representations that better reflect surface-level color perception, whereas language-supervised models exhibit a bias toward foreground object colors.

Abstract

Do vision models see colors the way humans do? Existing evaluations of color representations usually compare them with geometric spaces such as CIELAB or with discrete color labels. These references capture perceptual distance or category membership, but not the graded way in which people organize colors. We evaluate color grounding against a fuzzy perceptual model with 86 graded categories fitted to human survey data. The framework can be applied to any image encoder and measures three complementary properties: category boundaries, category compactness, and graded alignment beyond what color geometry alone can explain. Across eleven Vision Transformer encoders, the category-level results are broadly similar, whereas graded alignment differs substantially. Masked Autoencoders achieve the strongest beyond-geometry alignment, with confidence intervals that do not overlap those of the other encoders. A layer-wise analysis further shows that masked reconstruction preserves this structure toward the output. On natural images, MAE represents surface color globally, while language-supervised models encode color more strongly in relation to the foreground object. These results show that human-like color grounding has several distinct aspects that should not be reduced to a single score.

Problem

• color representation
• human perception
• vision models
• graded categories
• color grounding

Innovation

• graded color categories
• color grounding
• masked autoencoders
• human-like color perception
• beyond-geometry alignment