A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair

2026-04-23

AI Summary

This work addresses the inflated performance of large language models (LLMs) in automated program repair due to memorization effects arising from overlaps between training data and evaluation benchmarks. To systematically assess this issue, the authors propose a novel approach that integrates metamorphic testing with negative log-likelihood (NLL) metrics, constructing mutated benchmarks through semantics-preserving code transformations. Evaluations on Defects4J and GitBug-Java across seven prominent LLMs reveal a significant drop in repair success rates on the transformed benchmarks—ranging from −4.1% to −15.98%—with the magnitude of decline strongly correlated with NLL scores. These findings provide robust empirical evidence of widespread memorization and offer an effective methodology for detecting and mitigating evaluation bias in program repair tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

LLM-based automated program repair (APR) techniques have shown promising results in reducing debugging costs. However, prior results can be affected by data leakage: large language models (LLMs) may memorize bug fixes when evaluation benchmarks overlap with their pretraining data, leading to inflated performance estimates. In this paper, we investigate whether we can better reveal data leakage by combining metamorphic testing (MT) with negative log-likelihood (NLL), which has been used in prior work as a proxy for memorization. We construct variant benchmarks by applying semantics-preserving transformations to two widely used datasets, Defects4J and GitBug-Java. Using these benchmarks, we evaluate the repair success rates of seven LLMs on both original and transformed versions, and analyze the relationship between performance degradation and NLL. Our results show that all evaluated state-of-the-art LLMs exhibit substantial drops in patch generation success rates on transformed benchmarks, ranging from -4.1% for GPT-4o to -15.98% for Llama-3.1. Furthermore, we find that this degradation strongly correlates with NLL on the original benchmarks, suggesting that models perform better on instances they are more likely to have memorized. These findings show that combining MT with NLL provides stronger and more reliable evidence of data leakage, while metamorphic testing alone can help mitigate its effects in LLM-based APR evaluations.

Problem

• data leakage
• memorization
• large language models
• automated program repair
• evaluation bias

Innovation

• Metamorphic Testing
• Negative Log-Likelihood
• Data Leakage
• LLM-based Program Repair
• Memorization Diagnosis