Can LLMs be Effective Code Contributors? A Study on Open-source Projects

2026-04-25

AI Summary

This study addresses the significant limitations of current large language models (LLMs) in generating production-ready code for open-source projects, including high error rates, weak contextual understanding, and overreliance on training data. It presents the first systematic evaluation of mainstream LLMs—GPT-4o, Mistral3, and Qwen3-Coder—on their ability to fix bugs and implement new features across eight prominent open-source projects. The authors introduce a comprehensive evaluation framework integrating static verification, project-native test suites, and context-aware code generation. Experimental results reveal that LLM success rates on real-world tasks range from 0% to 60%, with models generally struggling to produce correct new code and exhibiting limited capacity in handling large-scale context. The work further identifies key failure modes and outlines targeted directions for improvement.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

LLM-generated code is widely used, and the share of committed code produced by LLMs is expected to increase. However, we are not at a point where LLMs can be effective contributors to production code. We present an approach that exposes the shortcomings of LLM generation on such projects, and proposes recommendations; the targets of our study are sizable open-source projects, e.g., FFmpeg and wolfSSL. First, we developed a framework that uses verification and validation to evaluate a given LLM's suitability to fix or add features to an existing project. Second, we apply the framework to 212 commits (bug fixes and small feature improvements) in eight popular open-source projects and three LLMs: GPT-4o, Ministral3, and Qwen3-Coder. The success rate varied from 0% to 60% depending on the project. The LLMs failed in a variety of ways, from generating syntactically incorrect code, to producing code that fails basic (static) verification, or validation via the project's test suite. In particular, the LLMs struggle with generating new code, handling contexts (function or file) outside a certain size range, and in many cases their success is due to parroting code changes they have been trained on.

Problem

• LLM-generated code
• open-source projects
• code contribution
• code validation
• production code

Innovation

• LLM code generation
• verification and validation
• open-source software
• empirical evaluation
• code contribution