Spiking the training data to correct for test set contamination

2026-05-23

AI Summary

This work addresses the problem of inflated evaluation scores caused by test set contamination by proposing a novel correction method based on actively introducing controlled contamination—referred to as “spiking”—rather than merely detecting it. By injecting a small number of known test samples into the training set, the approach leverages Platt-scaled membership inference signals to construct predictors of memorization and correctness, which are then integrated with statistical estimation to calibrate contaminated performance scores. The study introduces the first simulation-based evaluation framework grounded in the Hubble model, demonstrating that the proposed calibration estimator substantially outperforms naive estimation. Notably, effective calibration is achievable with only around ten spike samples, and the calibrator exhibits strong cross-dataset transferability.

Abstract

The literature on test set contamination largely focuses on detection, but the correction of contaminated test scores is underexplored. Our core proposal is to spike the training data by intentionally contaminating some test examples at known rates. The spiked examples can then be used to calibrate predictors of model memorization which enable principled statistical correction of inflated test scores. To evaluate different correction estimators, we first present a simulation framework based on the Hubble models. Hubble models come in minimal pairs, where the perturbed model was deliberately contaminated with several test sets, while the standard model was not, serving as the counterfactual and correction target. We consider estimators that use information from a memorization predictor, correctness predictor, or both. In simulation, we establish basic statistical intuitions and show that estimators leveraging memorization and correctness information are better than naive estimation which makes no correction at all. We then instantiate several memorization and correctness predictors, and find that simple predictors such as Platt-scaled membership inference metrics provide good signal for correction. Finally, we examine the practical considerations of spiking. Simple memorization predictors need no more than 10 examples for calibration and often transfer from one dataset to another. Taken together, spiking is a promising solution for test set contamination.

Problem

• test set contamination
• score correction
• model memorization
• calibration
• evaluation bias

Innovation

• test set contamination
• spiking
• memorization predictor
• statistical correction
• membership inference