A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions

2026-05-25

AI Summary

This study addresses the lack of systematic understanding regarding the practical efficacy, integration challenges, and long-term implications of large language models (LLMs) in code generation tasks. Integrating the SWEBOK and HELM frameworks, the authors conduct a three-tiered systematic literature review—including snowballing search, quality appraisal, and inter-coder reliability assessment—to synthesize findings from 30 secondary studies published between 2017 and 2025. The analysis reveals that while LLMs perform well on benchmark evaluations, they exhibit limited generalization, poor robustness, and constrained efficiency in real-world settings, with risks related to bias and toxicity significantly underestimated. The paper calls for the development of domain-aware models and the establishment of standardized evaluation protocols to guide future research and practical deployment.

Abstract

Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While reported results are promising, the broader effects of such application and their integration into real-world development remain insufficiently understood with existing tertiary studies provide little in this area. Objective. This tertiary study consolidates secondary evidence on LLM-based CGTs, synthesizing the publication landscape, effects, scenarios, integration challenges, and future research directions. Method. Following systematic review guidelines, we searched in related digital libraries, complemented by backward-and-forward snowballing and screening step. Study quality was assessed and extraction reliability was audited with inter-rater agreement statistics. Evidence was synthesized using SWEBOK knowledge areas and the HELM framework. Results. We identify 30 secondary studies published between 2017-2025, with rapid growth since 2023. Accuracy seems strong on benchmarks but weakly supported for real-world generalization; robustness is fragile across tasks and configurations; efficiency constraints are pervasive; toxicity and bias are under-reported. Dominant challenges concern economic feasibility, evaluation validity, and socio-technical integration. Future directions suggest domain-aware model improvement and the need for holistic, standardized evaluation. Conclusion. LLM-based CGTs represent a fast-maturing yet unevenly evaluated research area, highlighting the need for domain-aware model improvements and holistic, standardized evaluation, addressing efficiency and associated costs.

Problem

• large language models
• code generation
• software engineering
• tertiary review
• real-world integration

Innovation

• tertiary review
• large language models
• code generation
• systematic synthesis
• evaluation framework