Are LLM-based Chatbots Good Enough to Support Computer Science Students in Multiple-Choice Exercises?

2026-06-14

AI Summary

This study systematically evaluates the efficacy of large language model (LLM) chatbots in assisting university students with multiple-choice questions (MCQs) in computer science courses and investigates their actual impact on student learning. Drawing on a set of 70 course-aligned questions from interactive visualization and computer vision classes, we compare the performance of state-of-the-art models—including GPT-4o and GPT-5—under various prompting strategies and conduct user studies to assess how model-generated answers and explanations influence student accuracy. Results demonstrate that advanced LLMs significantly outperform smaller counterparts; however, providing students with these model outputs does not consistently improve their performance. This work provides the first empirical evidence delineating the capability boundaries and pedagogical utility of LLMs in higher education MCQ contexts.

Abstract

Chatbots based on large language models (LLMs) are increasingly adopted for information retrieval, text generation, and writing assistance. In educational settings, their use is also rapidly increasing. Students leverage these systems to complete tasks, access information, and support learning. However, the role of LLM-based chatbots in supporting learning and assessment in university-level computer science education is still underexplored. To address this gap, we investigate the performance of several LLM-based chatbots in solving multiple-choice questions (MCQs) at the university level and evaluate their capabilities to assist student learning. We developed 70 MCQs for a university lecture on interactive visual data analysis and evaluated the chatbots' performance using different prompt designs. We further compared the results with students' performance. Finally, we conducted a user study in two lectures (interactive visual data analysis, computer vision) to investigate how chatbot-generated answers and explanations affect students' performance. The chatbot performance showed significant differences between smaller models and GPT-4o and GPT-5 models, which achieved the best results. The results of the user study show that presenting ChatGPT answers together with an explanation does not improve students' performance in general.

Problem

• LLM-based chatbots
• computer science education
• multiple-choice questions
• student learning
• educational assessment

Innovation

• LLM-based chatbots
• multiple-choice questions
• computer science education
• prompt design
• user study