C2LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation

2024-12-06

arXiv.org

AI Summary

LLMs face severe credibility challenges in evaluation due to training data contamination—especially problematic given the inaccessibility of proprietary training corpora. To address this, we propose the first systematic contamination-free bilingual (Chinese–English) LLM evaluation benchmark, encompassing 22 fine-grained capability tasks. Our method introduces an automated contamination mitigation framework comprising: (i) a dynamic dataset rotation pipeline, (ii) a sensitive information filtering algorithm, (iii) a bilingual task design framework, and (iv) a standardized evaluation protocol—ensuring end-to-end privacy preservation across the evaluation data lifecycle. The resulting “bilingual, multidimensional, contamination-free” evaluation paradigm is rigorously validated on 15 open- and closed-weight LLMs. Empirical results demonstrate substantially improved evaluation integrity and reproducibility, establishing a new standard for fair, reliable, and comparable LLM assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data. To address this issue, we present C$^2$LEVA, a comprehensive bilingual benchmark featuring systematic contamination prevention. C$^2$LEVA firstly offers a holistic evaluation encompassing 22 tasks, each targeting a specific application or ability of LLMs, and secondly a trustworthy assessment due to our contamination-free tasks, ensured by a systematic contamination prevention strategy that fully automates test data renewal and enforces data protection during benchmark data release. Our large-scale evaluation of 15 open-source and proprietary models demonstrates the effectiveness of C$^2$LEVA.

Problem

• Addresses data contamination in LLM evaluation
• Provides comprehensive bilingual benchmark for LLMs
• Ensures trustworthy assessment with contamination-free tasks

Innovation

• Comprehensive bilingual benchmark for LLM evaluation
• Systematic contamination prevention strategy
• Automated test data renewal and protection