Understanding Student Perceptions, Mistakes, and Debugging Approaches when Solving Natural Language Programming Tasks

2026-07-06

AI Summary

This study investigates common error patterns, critical omissions, and debugging strategies employed by novice programmers when using natural language prompts for code generation. Drawing on behavioral data from over 900 CS1 students engaged in conversational programming tasks, the research integrates large-scale log analysis, qualitative content coding, and educational empirical methods to systematically uncover characteristic prompt deficiencies stemming from overreliance on AI reasoning capabilities. The findings reveal that beginners predominantly debug by clarifying their intent rather than tracing code logic. Additionally, students generally perceive prompt-based programming as more engaging and less demanding, reporting enhanced problem-solving confidence and competence as a result.

Abstract

Learning to communicate with code-generating AI models is an emerging skill for novice programmers. One recent pedagogical approach, Prompt Problems, has students solve computational tasks by writing natural-language prompts for code-generating AI models. However, little is known about the specific prompt-level mistakes novice programmers make, the kinds of computational details they fail to communicate, and what strategies they use to recover when generated code is incorrect. In a CS1 course, we studied attempts by more than 900 students to solve dialogue-based Prompt Problems. We analyzed student reflections, unsuccessful prompts, and reported debugging strategies. Compared to traditional coding tasks, students generally found prompting easier, more enjoyable, and better targeted at developing problem-solving skills. The most common mistakes are related to the omission of key details, suggesting both a failure to acknowledge their importance and over-reliance on AI to infer them. When prompts failed, students focused more on clarifying their intent and reflecting on the provided problem details than on tracing generated code or examining test cases.

Problem

• natural language programming
• prompt errors
• debugging strategies
• novice programmers
• code-generating AI

Innovation

• Prompt Problems
• natural-language programming
• code-generating AI
• novice programmers
• debugging strategies