CodeSwift: Accelerating LLM Inference for Efficient Code Generation

2025-02-24

AI Summary

To address the high autoregressive decoding latency of large language models (LLMs) in code generation and the neglect of code-specific syntactic and semantic characteristics by existing inference acceleration methods, this paper proposes the first lightweight, code-oriented inference acceleration framework. Our approach innovatively integrates three key components: (1) a multi-source structured knowledge base grounded in abstract syntax trees (ASTs) and semantic analysis; (2) a timing-aware dynamic retrieval mechanism; and (3) a context- and LLM-preference-aware caching strategy. Evaluated on repository-level and function-level code generation tasks, the framework achieves end-to-end speedups of 2.53× and 2.54×, respectively—outperforming state-of-the-art methods by up to 88%. Crucially, these gains are attained without compromising generation quality, thereby effectively balancing inference efficiency and functional correctness.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Code generation is a latency-sensitive task that demands high timeliness, but the autoregressive decoding mechanism of Large Language Models (LLMs) leads to poor inference efficiency. Existing LLM inference acceleration methods mainly focus on standalone functions using only built-in components. Moreover, they treat code like natural language sequences, ignoring its unique syntax and semantic characteristics. As a result, the effectiveness of these approaches in code generation tasks remains limited and fails to align with real-world programming scenarios. To alleviate this issue, we propose CodeSwift, a simple yet highly efficient inference acceleration approach specifically designed for code generation, without comprising the quality of the output. CodeSwift constructs a multi-source datastore, providing access to both general and project-specific knowledge, facilitating the retrieval of high-quality draft sequences. Moreover, CodeSwift reduces retrieval cost by controlling retrieval timing, and enhances efficiency through parallel retrieval and a context- and LLM preference-aware cache. Experimental results show that CodeSwift can reach up to 2.53x and 2.54x speedup compared to autoregressive decoding in repository-level and standalone code generation tasks, respectively, outperforming state-of-the-art inference acceleration approaches by up to 88%.

Problem

• Improves LLM inference efficiency
• Addresses code generation latency
• Incorporates syntax and semantics

Innovation

• Multi-source datastore for knowledge retrieval
• Parallel retrieval to enhance efficiency
• Context-aware cache for speed optimization