LLM-based vs. Search-based Merge Conflict Resolution: An Empirical Study of Competing Paradigms

2026-05-15

AI Summary

This study addresses the lack of systematic comparison between large language models (LLMs) and search-based software engineering (SBSE) approaches in resolving software merge conflicts. It presents the first empirical evaluation contrasting MergeGen, an LLM-based tool, with SBCR, an SBSE method employing random-restart hill climbing, on real-world merge conflict datasets from Java, C#, JavaScript, and TypeScript projects. The results reveal that LLMs excel at resolving content-imbalanced conflicts but are hindered by non-English contexts and long input sequences, whereas SBSE demonstrates superior generalization and outperforms LLMs on balanced conflicts. These findings highlight the complementary strengths of both paradigms and provide empirical grounding for developing hybrid merge conflict resolution strategies.

Abstract

Context: The resolution of software merge conflicts is being reshaped by two competing paradigms: generative approaches based on Large Language Models (LLMs) and optimization approaches from Search-Based Software Engineering (SBSE). While tools from both paradigms have shown promise, their relative strengths, weaknesses, and trade-offs are not yet well understood. Objective: This paper presents the first in-depth empirical study directly comparing these paradigms to identify their capabilities and limitations in real-world scenarios. Method: We evaluated MergeGen, a state-of-the-art LLM-based tool, against SBCR, a novel SBSE approach employing a Random Restart Hill Climbing (RRHC) algorithm. The comparison used thousands of real-world conflicts from open-source projects written in Java, C#, JavaScript, and TypeScript. Results: Our findings reveal fundamental trade-offs. The LLM paradigm excels at resolving conflicts with imbalanced content by leveraging learned patterns. However, it struggles with non-English content and large inputs, which can lead to truncated or empty resolutions. Conversely, the SBSE paradigm demonstrates superior generalization across datasets and performs best on balanced conflicts, highlighting its potential as a robust, data-independent alternative. Conclusions: Neither paradigm is a silver bullet. Our findings highlight context-dependent strengths and motivate the development of hybrid systems that combine the complementary capabilities of LLM and SBSE approaches to create more robust and reliable merge conflict resolution tools.

Problem

• merge conflict resolution
• Large Language Models
• Search-Based Software Engineering
• empirical comparison
• software merging

Innovation

• merge conflict resolution
• Large Language Models
• Search-Based Software Engineering
• empirical study
• hybrid approach