Uncertainty-based Debiasing and Unlearning for Decontamination

2026-06-22

AI Summary

This work addresses the pervasive issue of inflated performance in large language model (LLM) evaluations due to training data contamination. Existing decontamination approaches rely on clean reference models and focus solely on aggregate accuracy, failing to capture sample-level variations in contamination effects. To overcome these limitations, the authors propose an Uncertainty-Based Decontamination (UBD) method that estimates the degree of memorization for each test sample through deep ensembles of the contaminated model itself, without requiring any clean model. UBD constructs a debiased target distribution from these uncertainty estimates to guide either output calibration or parameter fine-tuning. Introducing the first sample-level decontamination evaluation framework—augmented with distributional distance metrics—the method significantly outperforms rewriting and option-permutation baselines on MMLU-Pro and MATH-MCQA, effectively mitigating contamination while preserving original performance on uncontaminated samples.

Abstract

Benchmark-based evaluation is the dominant paradigm for assessing large language model (LLM) capabilities, yet data contamination inflates reported performance and undermines fair comparison. Existing decontamination methods are evaluated solely through aggregate accuracy, which can obscure substantial differences in per-sample model behaviour, and many require access to an uncontaminated model. In this paper, we propose a sample-level evaluation framework for decontamination that complements accuracy-based assessment with distributional distance metrics, measuring how closely a decontaminated model recovers the output distribution of an uncontaminated model on each sample. Building on this framework, we introduce Uncertainty-Based Decontamination (UBD), a family of methods that leverage deep ensembles of the contaminated model to estimate per-sample memorization without requiring a uncontaminated model or knowledge of which samples are contaminated. UBD estimates a per-sample correction scalar from ensemble uncertainty, which is used to construct a debiased target distribution that suppresses the inflated probability mass on correct answers induced by contamination. This target is then used either as a post-hoc output correction (debiasing) or as a soft training signal for parameter update (unlearning). Experiments on MMLU-Pro and MATH-MCQA across multiple LLM backbones demonstrate that UBD produces per-sample output distributions substantially closer to those of an uncontaminated model than paraphrasing or choice-permutation baselines, while preserving model performance on uncontaminated data.

Problem

• data contamination
• decontamination
• large language models
• benchmark evaluation
• model debiasing

Innovation

• Uncertainty-Based Decontamination
• sample-level evaluation
• deep ensembles
• distributional distance
• model unlearning