Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models

2025-03-09

AI Summary

Static benchmarks for code large language models (LLMs) suffer from training data contamination, leading to inflated and unreliable performance estimates. Method: This paper introduces the first dynamic contamination-resistant benchmarking framework. It automatically generates syntactically novel yet semantically equivalent test cases via semantics-preserving program mutations—such as variable renaming and control-flow–equivalent transformations—followed by formal consistency verification to guarantee functional equivalence. Contribution/Results: Evaluated on ten state-of-the-art code LLMs, the framework induces substantial performance degradation (average drop >40%), revealing severe overestimation of model capabilities by static benchmarks and triggering significant rank reversals among models. This work pioneers dynamic benchmark reconstruction, establishing a new paradigm for trustworthy, contamination-aware evaluation of code LLMs.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them during training. We introduce a novel solution, dynamic benchmarking framework, to address this challenge. Given a code understanding or reasoning benchmark, our framework dynamically transforms each input, i.e., programs, with various semantic-preserving mutations to build a syntactically new while semantically identical benchmark. We evaluated ten popular language models on our dynamic benchmarks. Our evaluation reveals several interesting or surprising findings: (1) all models perform significantly worse than before, (2) the ranking between some models shifts dramatically, and (3) our dynamic benchmarks can resist against the data contamination problem.

Problem

• Dynamic benchmarking for code language models evaluation
• Addressing data contamination in model training benchmarks
• Semantic-preserving mutations to create new, useful benchmarks

Innovation

• Dynamic benchmarking framework for code models
• Semantic-preserving mutations on program inputs
• Resists data contamination in model evaluation