AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation

2025-11-16

International Conference on Automated Software Engineering

AI Summary

This work addresses the lack of effective evaluation benchmarks for large language models (LLMs) in code snippet adaptation tasks, which hinders a deeper understanding of their utility in software reuse. To bridge this gap, we introduce CAPT, the first benchmark specifically designed for code adaptation, constructed from real-world developer practices by integrating data from Stack Overflow and GitHub. CAPT features a dual-layer annotation scheme—task-level and adaptation-level—and a joint adaptation-function testing framework, enabling fine-grained assessment of models’ instruction-following and functional implementation capabilities. We evaluate six instruction-tuned and three reasoning-focused LLMs, revealing significant deficiencies in current models’ ability to follow explicit adaptation instructions. Our benchmark provides both critical insights and practical tools to advance future research in this domain.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advancements in large language models (LLMs) have automated various software engineering tasks, with benchmarks emerging to evaluate their capabilities. However, for adaptation, a critical activity during code reuse, there is no benchmark to assess LLMs’ performance, leaving their practical utility in this area unclear. To fill this gap, we propose AdaptEval, a benchmark designed to evaluate LLMs on code snippet adaptation. Unlike existing benchmarks, AdaptEval incorporates the following three distinctive features: First, practical context. Tasks in AdaptEval are derived from developers’ practices, preserving rich contextual information from Stack Overflow and GitHub communities. Second, multi-granularity annotation. Each task is annotated with requirements at both task and adaptation levels, supporting the evaluation of LLMs across diverse adaptation scenarios. Third, fine-grained evaluation. AdaptEval includes a two-tier testing framework combining adaptation-level and function-level tests, which enables evaluating LLMs’ performance across various individual adaptations. Based on AdaptEval, we conduct the first empirical study to evaluate six instruction-tuned LLMs and especially three reasoning LLMs on code snippet adaptation. Experimental results demonstrate that AdaptEval enables the assessment of LLMs’ adaptation capabilities from various perspectives. It also provides critical insights into their current limitations, particularly their struggle to follow explicit instructions. We hope AdaptEval can facilitate further investigation and enhancement of LLMs’ capabilities in code snippet adaptation, supporting their real-world applications.

Problem

• code snippet adaptation
• large language models
• benchmark
• code reuse
• software engineering

Innovation

• code snippet adaptation
• benchmark
• large language models
• multi-granularity annotation
• fine-grained evaluation