Shape Bias and Robustness Evaluation via Cue Decomposition for Image Classification and Segmentation

2025-03-16

AI Summary

This work addresses the shape-versus-texture bias in deep neural networks for image classification and semantic segmentation, along with the challenge of robustness evaluation under image degradations. We propose the first AI-free shape–texture disentanglement framework: it achieves model-agnostic cue separation via morphological contour extraction and texture-transfer filtering. We introduce the first shape-preference quantification metric tailored for semantic segmentation. Furthermore, we generalize this into a disentangled robustness metric that independently assesses each cue’s sensitivity to degradations. Our method is validated on ImageNet; on Cityscapes and ADE20K, we uncover—for the first time—a pronounced shape bias in segmentation models. The proposed robustness estimates significantly outperform existing baselines in accuracy.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Previous works studied how deep neural networks (DNNs) perceive image content in terms of their biases towards different image cues, such as texture and shape. Previous methods to measure shape and texture biases are typically style-transfer-based and limited to DNNs for image classification. In this work, we provide a new evaluation procedure consisting of 1) a cue-decomposition method that comprises two AI-free data pre-processing methods extracting shape and texture cues, respectively, and 2) a novel cue-decomposition shape bias evaluation metric that leverages the cue-decomposition data. For application purposes we introduce a corresponding cue-decomposition robustness metric that allows for the estimation of the robustness of a DNN w.r.t. image corruptions. In our numerical experiments, our findings for biases in image classification DNNs align with those of previous evaluation metrics. However, our cue-decomposition robustness metric shows superior results in terms of estimating the robustness of DNNs. Furthermore, our results for DNNs on the semantic segmentation datasets Cityscapes and ADE20k for the first time shed light into the biases of semantic segmentation DNNs.

Problem

• Evaluates shape and texture biases in DNNs for image classification and segmentation.
• Introduces a cue-decomposition method to extract shape and texture cues from images.
• Proposes a robustness metric to assess DNN performance against image corruptions.

Innovation

• AI-free data pre-processing for cue extraction
• Novel cue-decomposition shape bias evaluation metric
• Cue-decomposition robustness metric for DNNs