UNMASK: Discovering and Causally Verifying Spurious Shortcuts in Text Classifiers

2026-08-10

AI Summary

Neural text classifiers are highly susceptible to spurious correlations between labels and superficial patterns, leading to poor generalization on out-of-distribution or adversarial data. This work proposes UNMASK, the first end-to-end framework that operates without manual annotations by automatically generating interpretable Boolean features from unlabeled data, validating them through statistical and causal counterfactual analyses, and leveraging the validation outcomes to construct unsupervised groups for reweighting during training. The method successfully recovers multiple known biases on MNLI, achieving up to a 12.58 percentage point improvement in accuracy on HANS. On CivilComments-WILDS, it matches the worst-group performance of manually annotated DFR while substantially enhancing model robustness.

Abstract

Neural language models trained on large crowdsourced corpora frequently exploit spurious surface patterns tied to target labels without true linguistic or causal relevance, boosting benchmark performance while failing on adversarial or out-of-distribution inputs. Existing approaches either require manual specification of the feature vocabulary or automate discovery only partially, leaving the gap between dataset-level correlation and model-level exploitation unaddressed. We present U N M ASK, a fully automated pipeline that discovers, causally verifies, and mitigates spurious correlations in text classifiers without additional human annotation. Given unlabeled training examples, U N M ASK generates candidate surface patterns as executable boolean expressions, filters them through a statistical validation protocol with independent replication, and establishes causal model dependence via verified counterfactual interventions. Causally confirmed features then serve as annotation-free group definitions for Deep Feature Reweighting, eliminating the group labels that standard DFR requires. Applied to BERT and RoBERTa trained on MNLI, our pipeline independently rediscovers established lexical-overlap and negation biases, verifying 9 of 10 features on BERT and 6 on RoBERTa, and improving HANS accuracy by up to 12.58 pp. On CivilComments-WILDS, programmatic groups match the 70.1% worst- group accuracy of hand-labeled DFR (Kirichenko et al., 2023) without demographic annotation. We further demonstrate that the discovery and validation stages generalize to reward model preference data, surfacing interpretable spurious correlations in RewardBench2.

Problem

• spurious correlations
• text classifiers
• causal verification
• surface patterns
• distributional robustness

Innovation

• spurious correlations
• causal verification
• counterfactual intervention
• automated discovery
• Deep Feature Reweighting