eXplaining to Learn (eX2L): Regularization Using Contrastive Visual Explanation Pairs for Distribution Shifts

2026-05-07

AI Summary

Existing methods often exhibit unstable performance under distribution shifts, struggle to surpass empirical risk minimization baselines, and lack interpretability. To address these limitations, this work proposes an explanation-based regularization framework that explicitly decouples label-relevant features from confounding attributes at the group level. The approach constructs explanation pairs by comparing Grad-CAM activation maps generated by a main classifier and a confounder classifier, then introduces a similarity penalty to enforce disentanglement between class-discriminative and spurious features, thereby promoting functional domain invariance. Evaluated on the Spawrious Many-to-Many Hard Challenge, the method achieves an average accuracy of 82.24% and a worst-group accuracy of 66.31%, outperforming the current state of the art by 5.49% and 10.90%, respectively, while simultaneously enhancing both model performance and interpretability.

Abstract

Despite extensive research into mitigating distribution shifts, many existing algorithms yield inconsistent performance, often failing to outperform baseline Empirical Risk Minimization (ERM) across diverse scenarios. Furthermore, high algorithmic complexity frequently limits interpretability and offers only an indirect means of addressing spurious correlations. We propose eXplaining to Learn (eX2L): an interpretable, explanation-based framework that decorrelates confounding features from a classifier's latent representations during training. eX2L achieves this by penalizing the similarity between Grad-CAM activation maps generated by a primary label classifier and those from a concurrently trained confounder classifier. On the rigorous Spawrious Many-to-Many Hard Challenge benchmark, eX2L achieves an average accuracy (AA) of 82.24% +/- 3.87% and a worst-group accuracy (WGA) of 66.31% +/- 8.73%, outperforming the current state-of-the-art (SOTA) by 5.49% and 10.90%, respectively. Beyond its competitive performance, eX2L demonstrates that functional domain invariance can be achieved by explicitly decoupling label and nuisance attributes at the group level.

Problem

• distribution shifts
• spurious correlations
• domain invariance
• interpretability
• empirical risk minimization

Innovation

• contrastive visual explanations
• distribution shift
• spurious correlations
• Grad-CAM regularization
• domain invariance