Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets

2025-03-21

AI Summary

This study addresses fragmented task definitions, inconsistent evaluation protocols, and dataset biases hindering systematic progress in applying large language models (LLMs) to source code analysis. We conduct a systematic review of 217 publications from 2019–2024 and propose the first three-dimensional knowledge graph—spanning *tasks*, *models*, and *datasets*—specifically for code analysis, covering 12 analytical task categories, 8 mainstream LLM variants, and 19 core benchmarks. Methodologically, we introduce a reproducible, standardized evaluation framework grounded in empirical analysis. Through bibliometric analysis and architectural feature modeling, we identify key evolutionary trends and persistent bottlenecks—including limited generalizability and inadequate contextual modeling. Our contributions provide a structured conceptual foundation and methodological guidance for advancing LLM-driven code intelligence research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: (Computational) Cognitive Architectures

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) and transformer-based architectures are increasingly utilized for source code analysis. As software systems grow in complexity, integrating LLMs into code analysis workflows becomes essential for enhancing efficiency, accuracy, and automation. This paper explores the role of LLMs for different code analysis tasks, focusing on three key aspects: 1) what they can analyze and their applications, 2) what models are used and 3) what datasets are used, and the challenges they face. Regarding the goal of this research, we investigate scholarly articles that explore the use of LLMs for source code analysis to uncover research developments, current trends, and the intellectual structure of this emerging field. Additionally, we summarize limitations and highlight essential tools, datasets, and key challenges, which could be valuable for future work.

Problem

• Exploring LLMs' role in source code analysis tasks
• Investigating models and datasets for code analysis
• Identifying challenges in LLM-based code analysis workflows

Innovation

• LLMs for code analysis tasks
• Transformer-based architectures in workflows
• Summarizing tools, datasets, challenges