Execution Guided Line-by-Line Code Generation

2025-06-12

AI Summary

This paper addresses the underutilization of real-time execution feedback in neural code generation by proposing an execution-guided line-by-line generation framework. Methodologically: (1) it introduces a novel fine-grained execution signal injection mechanism that dynamically incorporates test-case execution results at each line generation step, ensuring cross-token consistency and line-level signal refreshment; (2) it designs a multi-agent parallel exploration architecture to enable task-level parallelism and diverse search over the solution space; (3) it integrates execution-guided classifier-free guidance (EG-CFG), dynamic prompt enhancement, and beam search sampling. Evaluated across tasks ranging from basic programming to competitive programming, the framework consistently outperforms strong baselines, achieving state-of-the-art performance on multiple metrics—particularly in executable rate and functional correctness.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present a novel approach to neural code generation that incorporates real-time execution signals into the language model generation process. While large language models (LLMs) have demonstrated impressive code generation capabilities, they typically do not utilize execution feedback during inference, a critical signal that human programmers regularly leverage. Our method, Execution-Guided Classifier-Free Guidance (EG-CFG), dynamically incorporates execution signals as the model generates code, providing line-by-line feedback that guides the generation process toward executable solutions. EG-CFG employs a multi-stage process: first, we conduct beam search to sample candidate program completions for each line; second, we extract execution signals by executing these candidates against test cases; and finally, we incorporate these signals into the prompt during generation. By maintaining consistent signals across tokens within the same line and refreshing signals at line boundaries, our approach provides coherent guidance while preserving syntactic structure. Moreover, the method naturally supports native parallelism at the task level in which multiple agents operate in parallel, exploring diverse reasoning paths and collectively generating a broad set of candidate solutions. Our experiments across diverse coding tasks demonstrate that EG-CFG significantly improves code generation performance compared to standard approaches, achieving state-of-the-art results across various levels of complexity, from foundational problems to challenging competitive programming tasks. Our code is available at: https://github.com/boazlavon/eg_cfg

Problem

• Incorporates real-time execution feedback into code generation
• Guides line-by-line code generation using execution signals
• Improves performance across diverse coding tasks

Innovation

• Incorporates real-time execution feedback during generation
• Uses multi-stage beam search and test execution
• Supports parallel agents for diverse solution exploration