MIGRATION-BENCH: Repository-Level Code Migration Benchmark from Java 8

2025-05-14

AI Summary

This work addresses the lack of systematic evaluation of large language models (LLMs) on Java cross-version code migration (Java 8 → 17/21). We introduce JVMigBench—the first repository-level benchmark for this task—comprising 5,102 real-world open-source projects and a curated subset of 300 high-complexity repositories. We propose the first fine-grained, repository-level, production-ready migration evaluation paradigm. To enable end-to-end migration decision-making and repair, we design SD-Feedback, a self-debugging strategy integrating LLM prompting (e.g., Claude-3.5-Sonnet-v2), static analysis, and iterative feedback. On the 300-project subset, SD-Feedback achieves 62.33% pass@1 success rate under minimal migration and 27.00% under maximal migration—marking substantial improvements in the assessability and practical utility of LLMs for industrial-scale code evolution tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

With the rapid advancement of powerful large language models (LLMs) in recent years, a wide range of software engineering tasks can now be addressed using LLMs, significantly enhancing productivity and scalability. Numerous benchmark datasets have been developed to evaluate the coding capabilities of these models, while they primarily focus on problem-solving and issue-resolution tasks. In contrast, we introduce a new coding benchmark MIGRATION-BENCH with a distinct focus: code migration. MIGRATION-BENCH aims to serve as a comprehensive benchmark for migration from Java 8 to the latest long-term support (LTS) versions (Java 17, 21), MIGRATION-BENCH includes a full dataset and its subset selected with $5,102$ and $300$ repositories respectively. Selected is a representative subset curated for complexity and difficulty, offering a versatile resource to support research in the field of code migration. Additionally, we provide a comprehensive evaluation framework to facilitate rigorous and standardized assessment of LLMs on this challenging task. We further propose SD-Feedback and demonstrate that LLMs can effectively tackle repository-level code migration to Java 17. For the selected subset with Claude-3.5-Sonnet-v2, SD-Feedback achieves 62.33% and 27.00% success rate (pass@1) for minimal and maximal migration respectively. The benchmark dataset and source code are available at: https://huggingface.co/collections/AmazonScience and https://github.com/amazon-science/self_debug respectively.

Problem

• Evaluates LLMs for Java 8 to LTS version migration
• Provides benchmark dataset for repository-level code migration
• Introduces SD-Feedback to improve migration success rates

Innovation

• Introduces MIGRATION-BENCH for Java 8 to LTS migration
• Uses SD-Feedback for repository-level code migration
• Provides comprehensive evaluation framework for LLMs