Web vs. LLMs: An Empirical Study of Learning Behaviors of CS2 Students

2025-01-21

AI Summary

This study investigates how second-year computer science students learn programming concepts using large language models (LLMs) such as ChatGPT versus traditional web resources (e.g., educational websites, tutorial videos). Employing a mixed-methods, within-subjects design, it integrates online learning tasks, screen recordings, surveys, semi-structured interviews, and behavioral log analysis. Results reveal that students interacting with LLMs exhibit distinct behavioral patterns: they formulate more explicit, direct queries and engage in significantly less iterative probing or follow-up questioning—indicating reduced depth of conceptual engagement. Conversely, for high-difficulty concepts, traditional resources yield superior conceptual mastery. The core contribution is the empirical identification of behaviorally mediated “transfer effects” in LLM-assisted learning—including query simplification and diminished interactive depth—and their differential impact on learning outcomes. These findings provide mechanistic insights into AI-augmented programming education and inform evidence-based pedagogical design.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

LLMs such as ChatGPT have been widely adopted by students in higher education as tools for learning programming and related concepts. However, it remains unclear how effective students are and what strategies students use while learning with LLMs. Since the majority of students' experiences in online self-learning have come through using search engines such as Google, evaluating AI tools in this context can help us address these gaps. In this mixed methods research, we conducted an exploratory within-subjects study to understand how CS2 students learn programming concepts using both LLMs as well as traditional online methods such as educational websites and videos to examine how students approach learning within and across both scenarios. We discovered that students found it easier to learn a more difficult concept using traditional methods than using ChatGPT. We also found that students ask fewer follow-ups and use more keyword-based queries for search engines while their prompts to LLMs tend to explicitly ask for information.

Problem

• Programming Education
• Super Language Assistants
• Learning Effectiveness

Innovation

• Empirical Study
• AI-Assisted Learning
• Behavioral Differences