Evaluating LLMs on Java Code Snippet Adaptation Using a Mutation-Injection Framework

2026-06-26

AI Summary

Existing benchmarks struggle to evaluate large language models’ ability to adapt code in the absence of explicit instructions, across multiple change types, and at the fragment-level granularity. This work proposes a mutation-injection framework based on open-source Java code, introducing— for the first time at the fragment level—a taxonomy of adaptation operations inspired by real developer behaviors. By leveraging controlled mutations and reinserting test suites, the framework assesses models’ contextual adaptation capabilities without requiring edit instructions. The approach supports multi-granular context control, enabling quantitative analysis of how different adaptation types affect model performance and revealing fundamental limits in scalability with respect to code complexity and contextual dependency.

Abstract

Background: Developers frequently reuse code by copying fragments and adapting them to fit new contexts. Existing benchmarks for evaluating large language models (LLMs) on code adaptation either rely on explicit step-by-step instructions, cover only narrow change types such as variable wiring, or operate exclusively at function-level granularity. It remains unknown how well LLMs can adapt code fragments without explicit edit guidance when the required changes are varied and controlled. Objective: We investigate instruction-free code snippet adaptation in which an LLM must adapt a code fragment to fit its target context without any explicit edit guidance. We study three dimensions: which adaptation types are hardest (RQ1), how performance scales with adaptation complexity (RQ2), and how much surrounding context the model needs (RQ3). Method: We will construct a dataset of Java code fragments from open-source repositories with strong test coverage and apply a taxonomy of adaptation operators, derived from empirical findings on how developers adapt copied code, using a mutation-injection framework. Working at the code fragment level and controlling the injected changes lets us know exactly what adaptations the model must perform. The unmutated fragment serves as a plausible reference for the changes the model needs to make. LLMs will be evaluated on instruction-free adaptation tasks across three context granularity levels. Correctness will be measured primarily via test-suite re-insertion, complemented by mutation-level inspection.

Problem

• code adaptation
• large language models
• instruction-free
• Java code snippets
• mutation-injection

Innovation

• mutation-injection
• instruction-free code adaptation
• code snippet granularity
• LLM evaluation
• test-suite re-insertion