AI-Assisted Code Review as a Scaffold for Code Quality and Self-Regulated Learning: An Experience Report

2026-04-25

AI Summary

This study addresses the challenges of scaling code review in software engineering education—namely, time constraints, inconsistent feedback, and students’ limited experience—by integrating large language models (LLMs) into the GitHub pull request (PR) workflow. The authors propose an in-workflow human-AI collaborative review mechanism that enables students to conduct authentic code reviews and fosters self-regulated learning. Through a controlled experiment across two course offerings, combining GitHub log data with student reflections, the study finds that the 2024 cohort submitted significantly more PRs (1,176 vs. 581), achieved zero AI invocation failures, and demonstrated substantive improvements in approximately one-third of AI-reviewed PRs. These results suggest students focused more on code quality discussions and reduced overreliance on AI, offering empirical support and design insights for AI-augmented code review pedagogy.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Code review is central to software engineering education but hard to scale in capstone projects due to tight deadlines, uneven peer feedback, and limited prior experience. We investigate an LLM-as-reviewer integrated directly into GitHub pull requests (human-in-the-loop) across two cohorts (more than 100 students, 2023--2024). Using a mixed-methods design -- GitHub data, reflective reports, and a targeted survey -- we examine engagement and responsiveness as behavioral indicators of self-regulated learning processes. Quantitatively, the 2024 cohort produced more iterative activity (1176 vs. 581 PRs), while technical issues observed in 2023 (227 failed AI attempts) dropped to zero after tool and instructional refinements. Despite different adoption levels (93\% vs. 50\% of teams using the tool), responsiveness was stable: 32\% (2023) and 33\% (2024) of successfully AI-reviewed PRs were followed by subsequent commits on the same PR. Qualitatively, students used the LLM's structured comments to focus reviews and discuss code quality, while guidance reduced over-reliance. We contribute: (i) an in-workflow design for an AI reviewer that scaffolds learning while mitigating cognitive offloading; (ii) a repeated cross sectional comparison across two cohorts in authentic settings; (iii) a mixed-methods analysis combining objective GitHub metrics with student self-reports; and (iv) evidence-based pedagogical recommendations for responsible, student-led AI-assisted review.

Problem

• code review
• software engineering education
• scalability
• peer feedback
• capstone projects

Innovation

• AI-assisted code review
• self-regulated learning
• LLM-as-reviewer
• GitHub pull requests
• cognitive offloading