Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming

2025-11-17

AI Summary

This study investigates how generative AI (e.g., ChatGPT) affects knowledge acquisition among programming learners of varying proficiency levels. Method: A controlled experiment was conducted, comparing generative AI with conventional online resources using task completion rates, conceptual understanding assessments, and fine-grained interaction log analysis. Contribution/Results: Generative AI significantly improves task completion for novices but yields inconsistent gains in conceptual knowledge; novices’ overreliance on AI-generated complete code impedes deep comprehension. In contrast, intermediate learners achieve superior learning outcomes through selective, strategic use. The study reveals a nonlinear relationship between usage patterns and learning efficacy, challenging the assumption that increased AI engagement uniformly enhances learning. Its core contribution is reframing generative AI as a cognitive scaffold—not merely a problem-solving tool—and proposing proficiency-adaptive design principles for AI-augmented programming instruction.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The rise of Generative AI (GenAI) tools like ChatGPT has created new opportunities and challenges for computing education. Existing research has primarily focused on GenAI's ability to complete educational tasks and its impact on student performance, often overlooking its effects on knowledge gains. In this study, we investigate how GenAI assistance compares to conventional online resources in supporting knowledge gains across different proficiency levels. We conducted a controlled user experiment with 24 undergraduate students of two different levels of programming experience (beginner, intermediate) to examine how students interact with ChatGPT while solving programming tasks. We analyzed task performance, conceptual understanding, and interaction behaviors. Our findings reveal that generating complete solutions with GenAI significantly improves task performance, especially for beginners, but does not consistently result in knowledge gains. Importantly, usage strategies differ by experience: beginners tend to rely heavily on GenAI toward task completion often without knowledge gain in the process, while intermediates adopt more selective approaches. We find that both over-reliance and minimal use result in weaker knowledge gains overall. Based on our results, we call on students and educators to adopt GenAI as a learning rather than a problem solving tool. Our study highlights the urgent need for guidance when integrating GenAI into programming education to foster deeper understanding.

Problem

• Comparing GenAI assistance with conventional online learning resources
• Investigating GenAI's impact on knowledge gains across programming proficiency levels
• Analyzing interaction behaviors and usage strategies with ChatGPT in programming education

Innovation

• Used controlled experiment with programming students
• Compared GenAI assistance to conventional online resources
• Analyzed task performance and knowledge gains