Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning system

2026-03-25

AI Summary

This study investigates how to effectively integrate adaptive feedback with generative AI–driven feedback to enhance code correctness and logical completeness among programming learners. We propose a hybrid approach that embeds large language models within an adaptive learning system, augmented by a retrieval-augmented generation (RAG) mechanism leveraging knowledge graphs and user interaction histories to deliver context-aware code evaluation, formative feedback, and personalized exercise recommendations. To our knowledge, this is the first systematic comparison in programming education of purely adaptive, purely generative AI–based, and hybrid instructional paradigms. Empirical results demonstrate that the hybrid model significantly increases the number of correct submissions, reduces logical omissions, and receives high learner satisfaction, thereby confirming its superiority and practical viability in intelligent tutoring systems.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper introduces the design and development of a framework that integrates a large language model (LLM) with a retrieval-augmented generation (RAG) approach leveraging both a knowledge graph and user interaction history. The framework is incorporated into a previously developed adaptive learning support system to assess learners' code, generate formative feedback, and recommend exercises. Moerover, this study examines learner preferences across three instructional modes; adaptive, Generative AI (GenAI), and hybrid GenAI-adaptive. An experimental study was conducted to compare the learning performance and perception of the learners, and the effectiveness of these three modes using four key log features derived from 4956 code submissions across all experimental groups. The analysis results show that learners receiving feedback from GenAI modes had significantly more correct code and fewer code submissions missing essential programming logic than those receiving feedback from adaptive mode. In particular, the hybrid GenAI-adaptive mode achieved the highest number of correct submissions and the fewest incorrect or incomplete attempts, outperforming both the adaptive-only and GenAI-only modes. Questionnaire responses further indicated that GenAI-generated feedback was widely perceived as helpful, while all modes were rated positively for ease of use and usefulness. These results suggest that the hybrid GenAI-adaptive mode outperforms the other two modes across all measured log features.

Problem

• adaptive feedback
• generative AI
• knowledge graph
• programming learning
• learning performance

Innovation

• Retrieval-Augmented Generation (RAG)
• Knowledge Graph
• Generative AI
• Adaptive Learning
• Hybrid Feedback System