Shortcut Learning Susceptibility in Vision Classifiers

2025-02-13

AI Summary

This work addresses the problem that visual classifiers—CNNs, MLPs, and Vision Transformers (ViTs)—often rely on spurious correlations (i.e., “shortcut learning”) in training data, degrading generalization and robustness. We propose the first unified framework to systematically evaluate the sensitivity of these three dominant architectures to *position-based shortcuts*. Methodologically, we inject controllable positional shortcuts into MNIST and CIFAR-10, and employ a dual-test-set evaluation protocol—comparing performance on standard versus shortcut-decoupled distributions—alongside gradient-based inverse reconstruction analysis to quantify and attribute shortcut reliance. Key findings: ViTs exhibit the highest vulnerability, while MLPs demonstrate the greatest robustness; all models show substantial shortcut dependence, which intensifies with increasing data complexity. This study provides empirical guidance for architecture selection and robust training, and introduces a novel analytical paradigm for diagnosing shortcut learning across model families.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Shortcut learning, where machine learning models exploit spurious correlations in data instead of capturing meaningful features, poses a significant challenge to building robust and generalizable models. This phenomenon is prevalent across various machine learning applications, including vision, natural language processing, and speech recognition, where models may find unintended cues that minimize training loss but fail to capture the underlying structure of the data. Vision classifiers such as Convolutional Neural Networks (CNNs), Multi-Layer Perceptrons (MLPs), and Vision Transformers (ViTs) leverage distinct architectural principles to process spatial and structural information, making them differently susceptible to shortcut learning. In this study, we systematically evaluate these architectures by introducing deliberate shortcuts into the dataset that are positionally correlated with class labels, creating a controlled setup to assess whether models rely on these artificial cues or learn actual distinguishing features. We perform both quantitative evaluation by training on the shortcut-modified dataset and testing them on two different test sets -- one containing the same shortcuts and another without them -- to determine the extent of reliance on shortcuts. Additionally, qualitative evaluation is performed by using network inversion-based reconstruction techniques to analyze what the models internalize in their weights, aiming to reconstruct the training data as perceived by the classifiers. We evaluate shortcut learning behavior across multiple benchmark datasets, including MNIST, Fashion-MNIST, SVHN, and CIFAR-10, to compare the susceptibility of different vision classifier architectures to shortcut reliance and assess their varying degrees of sensitivity to spurious correlations.

Problem

• Evaluates shortcut learning in vision classifiers.
• Assesses model reliance on spurious correlations.
• Compares susceptibility across multiple architectures.

Innovation

• controlled shortcut introduction
• quantitative and qualitative evaluation
• network inversion-based reconstruction