Calibrating Post-Training Feature Shifts for LLM Data Contamination Detection

2026-08-11

AI Summary

This work addresses the challenge that post-training techniques—such as instruction fine-tuning and preference optimization—induce characteristic shifts in large language model outputs, thereby degrading the separability between member and non-member texts in data contamination detection. To systematically mitigate this issue, the paper proposes CalibDCD, a novel calibration framework that, for the first time, identifies recurrent feature shifts through multi-perspective prompt variants and selectively alleviates their impact via a bounded feature correction mechanism while preserving effective detection signals. Extensive experiments demonstrate that CalibDCD consistently enhances the performance of diverse detectors, achieving up to a 7.0% absolute improvement in AUC and a 15.0% gain in TPR@5%FPR.

Abstract

Large language models (LLMs) are trained on massive and largely undisclosed corpora that may contain copyrighted or privacy-sensitive content. Data contamination detection (DCD) therefore aims to determine whether a given text is a member of the pre-training corpus of a target LLM. Recent state-of-the-art DCD methods follow a feature-based paradigm that derives membership features from the input text and the corresponding model output. However, most modern LLMs undergo post-training, such as instruction tuning, preference optimization, and reasoning-oriented training, which can alter model outputs and shift the corresponding membership features, thereby reducing the separability between members and non-members. To address this problem, we propose CalibDCD, a broadly applicable calibration framework for feature-based DCD methods, comprising (1) Multi-View Shift Detection, which identifies recurring feature shifts associated with post-training, and (2) Bounded Feature Correction, which selectively mitigates their influence on membership prediction. Specifically, Multi-View Shift Detection evaluates controlled prompt variants on known non-member texts and consolidates the most informative views to identify recurring feature shifts. Bounded Feature Correction selectively adjusts feature components aligned with the detected shifts and controls the correction extent to preserve useful detection information. Experiments show that CalibDCD consistently improves existing feature-based detectors, with gains of up to 7.0% in AUC and 15.0% in TPR@5%FPR.

Problem

• data contamination detection
• feature shift
• post-training
• large language models
• membership inference

Innovation

• CalibDCD
• feature shift calibration
• data contamination detection
• post-training adaptation
• membership inference