The Poisoned Chalice of LLM Evaluation Report

2026-07-08

AI Summary

This study addresses the challenge of inflated performance of large language models (LLMs) in software engineering evaluations due to pretraining data contamination, a problem exacerbated by the opacity of their training corpora. The work formalizes this issue as a white-box membership inference task over source code and introduces a general-purpose detection framework that operates across models and datasets. To catalyze progress in this area, the authors organized the first “Poisoned Cup” LLM evaluation competition (FSE-AIWare 2026), providing a standardized dataset, target models, and baseline methods. This initiative advances the development of generalizable decontamination-aware evaluation techniques and fosters a more trustworthy ecosystem for assessing LLMs in software engineering contexts.

Abstract

Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about whether benchmark instances were seen during pretraining. This can lead to data contamination, which may inflate performance and result in misleading conclusions about model capability. Despite this, the training corpora of many modern models are only partially disclosed, making direct decontamination infeasible. This creates a need for practical methods that can detect a large language models' prior exposure to training data without access to the full training corpus. To address this challenge, we organize the first Poisoned Chalice of LLM Evaluation Competition, co-located with the FSE-AIWare 2026 Competition Track. The competition frames contamination detection as a white-box membership inference task on source code and provides participants with curated datasets, target models, baseline attacks, and a final evaluation on a held-out model and dataset. This design encourages methods that generalize beyond superficial dataset artifacts and beyond a single training setting. This paper reports the setup and results of the competition. More broadly, the competition aims to catalyze the community around trustworthy LLM evaluation for software engineering.

Problem

• data contamination
• large language models
• membership inference
• software engineering
• evaluation validity

Innovation

• data contamination detection
• membership inference
• large language models
• software engineering evaluation
• white-box attack