Early Cue Precision Shapes Visual Shortcut Learning in Controlled Cue-Manipulation Benchmarks

2026-06-29

AI Summary

This study investigates how the accuracy of early visual cues—particularly texture—affects shortcut learning in image classification. By systematically manipulating object–texture correspondence across synthetic shape–texture tasks, sequential digit training, and a CIFAR-10 benchmark with superimposed textures, the authors demonstrate that low early-cue accuracy critically influences whether models rely on spurious correlations. Using frozen representation probing and multi-metric evaluation, they find that reduced texture fidelity mitigates performance collapse under cue–object conflict (e.g., CIFAR-10 conflict accuracy drops from 0.569 to 0.114 while texture selection rate rises to 0.855). However, this benefit is easily overridden by end-to-end fine-tuning. The work underscores that decorrelating cues from target labels must be actively maintained throughout downstream adaptation rather than addressed as a one-time preprocessing step.

Abstract

Visual classifiers can achieve high matched-distribution accuracy while relying on low-level cues that fail under conflict or suppression. We test whether this failure is shaped by early cue precision: the reliability with which a low-level cue predicts the label during early learning or downstream probe fitting. Across synthetic shape-texture tasks, sequential digit training, a 10-class frozen-representation audit, and a CIFAR-10 natural-image-based texture-overlay benchmark, we manipulate object-texture match probability and evaluate matched-ID accuracy, conflict accuracy, texture-choice rate, and suppression behavior. Degraded-but-predictive input does not substitute for cue decorrelation. In 10-class digit probes, conflict accuracy drops from 0.589 under chance-like cue precision to 0.005 under target-perfect texture. In CIFAR-10 frozen probes, conflict accuracy drops from 0.569 to 0.114, while texture choice rises from 0.049 to 0.855; this ordering persists across texture-overlay strengths alpha in {0.15,0.25,0.35,0.50}. End-to-end CIFAR-10 training shows that low early cue precision improves pre-target conflict behavior, but shortcut-rich fine-tuning can rapidly overwrite this benefit. Cue decorrelation must therefore be maintained during downstream adaptation rather than treated as a one-time inoculation.

Problem

• visual shortcut learning
• cue precision
• conflict accuracy
• texture bias
• downstream adaptation

Innovation

• early cue precision
• shortcut learning
• cue decorrelation
• conflict accuracy
• texture-overlay benchmark