Merge-Bench: Resolve Merge Conflicts with Large Language Models

2026-05-25

AI Summary

This work addresses the challenge of code merge conflicts in real-world software version control by proposing a fully automated, annotation-free construction methodology. The authors introduce MergeBench, a large-scale dataset comprising 7,938 real merge conflicts, and present LLMergeJ, a 14-billion-parameter language model trained using Group Relative Policy Optimization (GRPO)—a novel application of GRPO to large language model training for this task. Evaluated on Java conflict resolution, LLMergeJ outperforms three leading commercial large language models and approaches the performance of Gemini 2.5 Pro. Multilingual assessments further reveal that even state-of-the-art models achieve an overall resolution rate below 60%, underscoring both the inherent difficulty of merge conflict resolution and the pioneering nature of this research.

Abstract

This paper applies machine learning to the difficult and important task of version control merging. (1) We constructed a dataset, Merge-Bench, of 7938 real-world merge conflict hunks from 1439 GitHub repositories. The ground truth is the merge resolution that developers committed to the repository. Our dataset construction methodology is scalable to arbitrary amounts of data since no manual labeling is required. (2) We trained a model, LLMergeJ, to resolve merge conflicts in Java programs. Our approach uses Group Relative Policy Optimization (GRPO), an online reinforcement learning method, to train a Large Language Model (LLM). (3) We performed two evaluations of the performance of LLMs on resolving merge conflicts. On Java programs, LLMergeJ with 14B parameters outperforms 3 commercial LLMs, trailing only Gemini 2.5 Pro. Across 11 programming languages, commercial LLM performance is largely stable from language to language. The best models correctly resolve less than 60% of merge conflicts.

Problem

• merge conflicts
• version control
• code merging
• software development
• conflict resolution

Innovation

• Merge-Bench
• LLMergeJ
• Group Relative Policy Optimization
• merge conflict resolution
• large language models