Disagreement-Regularized Importance Sampling for Adversarial Label Corruption

2026-05-08

AI Summary

Standard importance sampling is highly susceptible to high-norm adversarial outliers under label corruption, leading to severe estimation bias. This work proposes a subsampling method that leverages disagreement in loss-based ranking among an ensemble of proxy models, introducing ranking disagreement as a novel regularizer within the importance sampling framework to effectively identify and filter contaminated samples. By integrating the ε-contamination model with finite-sample concentration analysis, the approach offers rigorous theoretical guarantees. Empirical evaluations on benchmark datasets demonstrate strong robustness against high-norm attacks, significantly outperforming magnitude-dependent methods such as EL2N and exhibiting complementary behavior to training-dynamics-based approaches like AUM.

Abstract

Standard Importance Sampling (IS) collapses under label corruption because high-norm examples, prioritized for variance reduction, are often adversarial outliers. We formalize this misalignment using an $\varepsilon$-contamination model and propose Disagreement-Regularized Importance Sampling (DR-IS), a sub-sampling method based on loss rank-disagreement across independent proxy ensemble. We prove finite-sample concentration bounds showing that the empirical rank disagreement of bulk corrupted examples is bounded above, and that of boundary-clean examples bounded below, both at rate $O(\sqrt{\log(N/δ)/K})$ with probability $1-δ$; when the structural expectation gap $Δ'$ between the two groups is positive and the boundary-clean set is at least as large as the selected subset, these bounds certify strict separation and control the contamination rate of the selected subset. Empirically, DR-IS remains robust under targeted high-norm attacks that break magnitude-based methods such as the Error $L_2$-norm (EL2N) on benchmark datasets. DR-IS complements training-dynamics approaches like Area Under the Margin ranking (AUM), offering improved robustness in the loss-aligned regime alongside explicit finite-sample concentration certificates and a contamination bound limiting noise leakage from the statistical tail of corrupted points.

Problem

• adversarial label corruption
• importance sampling
• outlier contamination
• robust subsampling
• high-norm attacks

Innovation

• Disagreement-Regularized Importance Sampling
• adversarial label corruption
• rank disagreement
• finite-sample concentration bounds
• proxy ensemble