Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation

2026-02-01

AI Summary

This work addresses the limitations of conventional large language models in code generation, which rely on monotonic, irreversible token-by-token decoding and thus fail to emulate the human-like iterative process of “writing while revising,” often leading to security vulnerabilities. To overcome this, the authors propose the “Stream of Revision” paradigm, which integrates learnable action tokens—such as backtracking, deletion, and rewriting—into autoregressive decoding. This enables the model to dynamically edit previously generated content within a single forward pass, effectively internalizing a revision loop. For the first time, code generation is extended from a static sequence to a self-correcting dynamic trajectory, activating the model’s intrinsic error-correction capability without external tools. The approach significantly reduces security vulnerabilities in generated code while maintaining minimal inference overhead.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Model (LLM) based code generation is predominantly formulated as a strictly monotonic process, appending tokens linearly to an immutable prefix. This formulation contrasts to the cognitive process of programming, which is inherently interleaved with forward generation and on-the-fly revision. While prior works attempt to introduce revision via post-hoc agents or external static tools, they either suffer from high latency or fail to leverage the model's intrinsic semantic reasoning. In this paper, we propose Stream of Revision, a paradigm shift that elevates code generation from a monotonic stream to a dynamic, self-correcting trajectory by leveraging model's intrinsic capabilities. We introduce specific action tokens that enable the model to seamlessly backtrack and edit its own history within a single forward pass. By internalizing the revision loop, our framework Stream of Revision allows the model to activate its latent capabilities just-in-time without external dependencies. Empirical results on secure code generation show that Stream of Revision significantly reduces vulnerabilities with minimal inference overhead.

Problem

• code generation
• autoregressive decoding
• on-the-fly revision
• secure code
• monotonic generation

Innovation

• Stream of Revision
• code generation
• self-correction
• action tokens
• secure coding