Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report

2026-08-12

AI Summary

This study addresses the challenges developers face when building large language model–based multi-agent systems, particularly in framework selection, agent role design, and coordination mechanisms. From a developer-centric perspective, the work presents the first systematic evaluation of prominent open-source multi-agent frameworks through a mixed-methods approach, combining quantitative analysis of documentation and functional capabilities with a qualitative README summarization task experiment evaluated using ROUGE metrics. The authors propose an integrated assessment framework encompassing functional coverage, documentation quality, and practical efficacy. Findings reveal that while existing frameworks support core components, they generally lack advanced features—such as agent telemetry—and exhibit no statistically significant performance differences in the summarization task. The study provides practitioners with an evidence-based framework selection guide, a checklist of key development challenges, and empirical insights to inform real-world deployment decisions.

Abstract

The proliferation of Generative Artificial Intelligence (Gen AI) powered by large language models (LLMs) has transformed the software development process, introducing new paradigms for code generation, debugging, testing, and maintenance. While early applications focused on leveraging single, independent LLMs to assist developers with isolated tasks, recent advances have shifted toward multi-agent systems (MAS) that orchestrate multiple LLM-based agents working collaboratively toward common objectives. Despite their promising potential, using MAS encompasses a set of challenges for developers who have to carefully select the right technology, devise proper coordination rules, and design specific roles for the involved agents. In this paper, we provide a comprehensive overview of the existing tools and frameworks for implementing MAS in software engineering. First, we conducted a quantitative analysis of the most relevant open source MAS frameworks by evaluating their documentation, features, and capabilities from the developers' perspective. Second, we performed a qualitative evaluation of a subset of the selected frameworks by implementing a common use case: the summarization of README.MD files. The findings show that the selected frameworks provide a good coverage of fundamental components of MAS, though advanced features such as telemetry of agents are still missing. In addition, the empirical evaluation shows that there is no significant difference in terms of ROUGE scores considering the summarization task. Finally, we provide a set of lessons learned and challenges that can help researchers and practitioners to select a suitable MAS framework according to their needs.

Problem

• Multi-Agent Systems
• Large Language Models
• Software Engineering
• Generative AI
• Framework Evaluation

Innovation

• Multi-Agent Systems
• Large Language Models
• Software Engineering
• Mixed-Methods Evaluation
• Gen AI