AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

2024-12-18

arXiv.org

AI Summary

Data contamination severely compromises the reliability of fair evaluation for large language models (LLMs), as test samples may have already appeared in model training corpora. Existing benchmark updates rely heavily on manual curation and struggle to prevent implicit knowledge leakage. To address this, we propose AntiLeak-Bench—the first benchmark framework integrating *knowledge timeliness verification* with *automated sample generation*. It rigorously aligns test content with LLMs’ official training cutoff dates, extracts real-time knowledge from diverse, time-stamped sources, synthesizes controlled textual stimuli, and deploys an end-to-end automation pipeline to generate test samples containing knowledge entirely absent from training sets. Empirical evaluation reveals pervasive pre-cutoff contamination in mainstream benchmarks. AntiLeak-Bench achieves zero manual maintenance and 100% contamination avoidance, thereby substantially enhancing evaluation objectivity, fairness, and reproducibility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.

Problem

• Preventing data contamination in LLM evaluation benchmarks
• Automating benchmark construction with updated real-world knowledge
• Ensuring contamination-free evaluation without human labor

Innovation

• Automated anti-leakage benchmarking framework
• Constructs samples with new knowledge
• Fully automated workflow without human labor