Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs

2026-04-15

AI Summary

This work addresses the opacity of training data in large language models for code, which complicates the assessment of data leakage risks. The authors propose a perturbation-based quantitative method to systematically measure a model’s “memorization advantage”—defined as the performance gap between inputs the model may have encountered during training and those it has not. By integrating perturbation testing, cross-model comparisons, and a multi-task benchmark encompassing code generation, comprehension, vulnerability detection, and repair, the study reveals that memorization advantage varies significantly with task type and model architecture. Experiments show that StarCoder exhibits notable memorization advantage on certain tasks, whereas QwenCoder demonstrates substantially less. Moreover, widely used datasets such as CVEFixes and Defects4J exert minimal memorization effects, suggesting that models primarily rely on generalization rather than rote memorization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The lack of transparency about code datasets used to train large language models (LLMs) makes it difficult to detect, evaluate, and mitigate data leakage. We present a perturbation-based method to quantify memorization advantage in code LLMs, defined as the performance gap between likely seen and unseen inputs. We evaluate 8 open-source code LLMs on 19 benchmarks across four task families: code generation, code understanding, vulnerability detection, and bug fixing. Sensitivity patterns vary widely across models and tasks. For example, StarCoder reaches high sensitivity on some benchmarks (up to 0.8), while QwenCoder remains lower (mostly below 0.4), suggesting differences in generalization behavior. Task categories also differ: code summarization tends to show low sensitivity, whereas test generation is substantially higher. We then analyze two widely discussed benchmarks, CVEFixes and Defects4J, often suspected of leakage. Contrary to common concerns, both show low memorization advantage across models: CVEFixes remains below 0.1, and Defects4J is lower than other program repair benchmarks. These results suggest that, for these datasets, models may rely more on learned generalization than direct memorization. Overall, our findings provide evidence that memorization risk is highly task- and model-dependent, and highlight the need for stronger evaluation protocols, especially in security-focused settings.

Problem

• memorization
• data leakage
• code LLMs
• generalization
• evaluation

Innovation

• memorization advantage
• code LLMs
• perturbation-based evaluation
• data leakage
• generalization behavior