Towards an Understanding of Context Utilization in Code Intelligence

2025-04-11

AI Summary

Code intelligence research has long lacked a systematic analysis of context utilization; although prior work confirms context improves model performance, context types remain ambiguously defined, integration strategies are fragmented, and evaluation protocols are inconsistent. Method: We conduct a comprehensive survey of 146 studies published between 2007 and 2024, introducing the first context taxonomy specifically designed for code intelligence, and establishing a three-dimensional analytical framework—“task–context–evaluation.” Using bibliometric analysis, systematic literature review, and qualitative coding, we rigorously synthesize empirical evidence. Contributions: (1) A quantitative developmental map of the field; (2) A novel, principled context classification system; (3) A comparative analysis of context integration strategies across 12 code intelligence tasks; and (4) A diagnostic assessment of prevalent evaluation shortcomings, accompanied by a practical, actionable research roadmap.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Community question answering

Abstract

Code intelligence is an emerging domain in software engineering, aiming to improve the effectiveness and efficiency of various code-related tasks. Recent research suggests that incorporating contextual information beyond the basic original task inputs (i.e., source code) can substantially enhance model performance. Such contextual signals may be obtained directly or indirectly from sources such as API documentation or intermediate representations like abstract syntax trees can significantly improve the effectiveness of code intelligence. Despite growing academic interest, there is a lack of systematic analysis of context in code intelligence. To address this gap, we conduct an extensive literature review of 146 relevant studies published between September 2007 and August 2024. Our investigation yields four main contributions. (1) A quantitative analysis of the research landscape, including publication trends, venues, and the explored domains; (2) A novel taxonomy of context types used in code intelligence; (3) A task-oriented analysis investigating context integration strategies across diverse code intelligence tasks; (4) A critical evaluation of evaluation methodologies for context-aware methods. Based on these findings, we identify fundamental challenges in context utilization in current code intelligence systems and propose a research roadmap that outlines key opportunities for future research.

Problem

• Lack of systematic analysis of context in code intelligence
• Need for understanding context integration strategies in code tasks
• Challenges in evaluating context-aware methods in code intelligence

Innovation

• Incorporating contextual information beyond source code
• Systematic analysis of context in code intelligence
• Novel taxonomy of context types used