From LLMs to Agents in Programming: The Impact of Providing an LLM with a Compiler

2026-01-17

AI Summary

This work proposes an intelligent programming agent that integrates large language models (LLMs) with the GCC compiler to iteratively refine generated code through compilation feedback. The approach addresses the prevalent issue of syntactic and linking errors in LLM-generated code, which often renders it non-compilable. A systematic evaluation across 699 C programming tasks from RosettaCode demonstrates that compiler feedback substantially improves code compilability: using 16 LLMs ranging from 135M to 70B parameters, compilation success rates increase by 5.3 to 79.4 percentage points, syntactic errors decrease by 75%, and undefined reference errors drop by 87%. Notably, smaller models augmented with compiler feedback can outperform larger, unassisted models, highlighting the significant potential of tool-augmented approaches for sustainable and efficient software development.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models have demonstrated a remarkable capability in natural language and program generation and software development. However, the source code generated by the LLMs does not always meet quality requirements and may fail to compile. Therefore, many studies evolve into agents that can reason about the problem before generating the source code for the solution. The goal of this paper is to study the degree to which such agents benefit from access to software development tools, in our case, a gcc compiler. We conduct a computational experiment on the RosettaCode dataset, on 699 programming tasks in C. We evaluate how the integration with a compiler shifts the role of the language model from a passive generator to an active agent capable of iteratively developing runnable programs based on feedback from the compiler. We evaluated 16 language models with sizes ranging from small (135 million) to medium (3 billion) and large (70 billion). Our results show that access to a compiler improved the compilation success by 5.3 to 79.4 percentage units in compilation without affecting the semantics of the generated program. Syntax errors dropped by 75%, and errors related to undefined references dropped by 87% for the tasks where the agents outperformed the baselines. We also observed that in some cases, smaller models with a compiler outperform larger models with a compiler. We conclude that it is essential for LLMs to have access to software engineering tools to enhance their performance and reduce the need for large models in software engineering, such as reducing our energy footprint.

Problem

• Large Language Models
• Code Generation
• Compiler Integration
• Programming Agents
• Software Development Tools

Innovation

• LLM agents
• compiler feedback
• iterative code generation
• software engineering tools
• model efficiency