SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models

2026-06-29

AI Summary

Existing approaches to detecting data leakage in large language models for code often rely on private training data, fragile heuristics, or non-generalizable thresholds, limiting their reliability and broad applicability. This work proposes SrDetection, the first self-referential leakage detection framework that operates without external reference sets or manually defined thresholds. By generating semantically equivalent code variants and analyzing discrepancies in the model’s outputs or logits between original and variant samples, SrDetection employs an adaptive measurement mechanism to effectively identify leakage under both gray-box and black-box settings. Experimental results demonstrate that SrDetection improves average F1 scores by 21.52 and 14.46 points over strong baselines in the two scenarios, respectively, and uncovers fine-grained leakage patterns across 15 prominent code large language models on four major benchmarks.

Abstract

Evaluating code large language models (Code LLMs) requires reliable detection of data leakage, where benchmark performance is artificially inflated by exposure to benchmark data during pre-training. Existing approaches either assume access to proprietary training corpora, rely on brittle heuristics such as timestamp filtering, or use external reference sets with manually tuned, non-generalizable thresholds. To address these limitations, we introduce \textbf{SrDetection}, a unified \textbf{s}elf-\textbf{r}eferential leakage detection framework for both gray-box (access to model logits) and black-box (access to model outputs) settings. SrDetection generates semantically equivalent variants of a benchmark sample and detects leakage by contrasting the model's behavior on the original versus its variants, flagging cases where the original is disproportionately easier for the model. We further design a controlled leakage detection testbed and evaluate SrDetection in this environment. Across different models and training stages, SrDetection improves average F1 by 21.52 points in the gray-box setting and 14.46 points in the black-box setting over strong baselines, demonstrating robust, threshold-independent leakage detection. Finally, a gray-box study of 15 widely used Code LLMs on four popular benchmarks reveals benchmark-specific leakage patterns beyond prior overlap-based analyses\footnote{\footnotesize Source code and data are available at https://github.com/SMinL/SrDetectionCode

Problem

• data leakage detection
• code large language models
• benchmark contamination
• self-referential framework
• model evaluation

Innovation

• self-referential
• data leakage detection
• code LLMs
• semantic variants
• threshold-independent