On the Influence of Shape, Texture and Color for Learning Semantic Segmentation

2024-10-18

arXiv.org

AI Summary

This study investigates the independent and synergistic effects of three visual cues—shape, texture, and color—on deep models (CNNs and Transformers) for semantic segmentation. We propose the first decoupled data generation framework enabling controlled manipulation of these cues and conduct pixel-level attribution analysis on Cityscapes, PASCAL Context, and CARLA. Our methodology includes cue decomposition, early fusion (mixed training), and late fusion (ensemble of cue-specialized models). Key findings are: (1) neither texture nor shape universally dominates segmentation performance; (2) the “shape + color” combination achieves near-full-cue accuracy even in the absence of texture, consistently across architectures; and (3) we provide the first systematic quantification of individual cue contributions, revealing that cue utilization patterns are highly architecture-agnostic. These results advance our understanding of representational mechanisms in segmentation models and establish a new paradigm for designing robust vision systems grounded in interpretable, cue-decomposed analysis.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In recent years, a body of works has emerged, studying shape and texture biases of off-the-shelf pre-trained deep neural networks (DNN) for image classification. These works study how much a trained DNN relies on image cues, predominantly shape and texture. In this work, we switch the perspective, posing the following questions: What can a DNN learn from each of the image cues, i.e., shape, texture and color, respectively? How much does each cue influence the learning success? And what are the synergy effects between different cues? Studying these questions sheds light upon cue influences on learning and thus the learning capabilities of DNNs. We study these questions on semantic segmentation which allows us to address our questions on pixel level. To conduct this study, we develop a generic procedure to decompose a given dataset into multiple ones, each of them only containing either a single cue or a chosen mixture. This framework is then applied to two real-world datasets, Cityscapes and PASCAL Context, and a synthetic data set based on the CARLA simulator. We learn the given semantic segmentation task from these cue datasets, creating cue experts. Early fusion of cues is performed by constructing appropriate datasets. This is complemented by a late fusion of experts which allows us to study cue influence location-dependent on pixel level. Our study on three datasets reveals that neither texture nor shape clearly dominate the learning success, however a combination of shape and color but without texture achieves surprisingly strong results. Our findings hold for convolutional and transformer backbones. In particular, qualitatively there is almost no difference in how both of the architecture types extract information from the different cues.

Problem

• Analyzing shape, texture and color cue influences during semantic segmentation training
• Investigating individual and combined cue impacts on DNN learning success
• Evaluating cue performance across architectures for object boundary prediction

Innovation

• Analyzes cue influence by decomposing datasets
• Uses early fusion by constructing reduced cue datasets
• Performs late fusion of experts at pixel level