Comparing Large Language Models on Scrum Certification-Style Questions: Accuracy, Stability, and Error Patterns

2026-06-29

AI Summary

This study systematically evaluates the accuracy, stability, and error patterns of large language models in answering normative questions related to Scrum certification. Drawing on a dataset of 993 Scrum-style items, the authors conduct multiple rounds of comparative experiments with GPT-5 Mini, Gemini 3 Flash, and DeepSeek Chat 3.2 using three prompting strategies: zero-shot, chain-of-thought, and source-grounded reasoning. Combining quantitative and qualitative analyses, the work reveals systematic errors in how these models interpret Scrum guidelines. Gemini 3 Flash demonstrates superior performance overall. While models achieve high accuracy on single-choice questions, they struggle with multiple-select and true/false formats. Performance remains robust on well-defined topics but degrades significantly on abstract dimensions such as values and team collaboration. The primary sources of error include overgeneralization and conflation of Scrum’s official definitions with prevailing market practices.

Abstract

Large Language Models (LLMs) are increasingly used in exam- and certification-style question answering tasks, where their ability to retrieve, interpret, and apply domain-specific knowledge can be systematically assessed. In Software Engineering, such settings are particularly relevant when questions depend on strict adherence to normative definitions, roles, artifacts, and rules. This paper evaluates the performance of three contemporary LLMs, \textit{GPT-5 mini}, \textit{Gemini 3 Flash}, and \textit{DeepSeek Chat 3.2}, in answering 993 Scrum certification-style questions aligned with the Professional Scrum Master I (PSM I) assessment format. We evaluated the models under three prompting strategies (\textit{zero-shot}, \textit{chain-of-thought}, and \textit{source-grounded}), with repeated executions to assess intra-model stability. We also analyzed performance across Scrum topics and question formats, complemented by a qualitative analysis of recurring error patterns in incorrect answers. Results revealed clear differences among models, with Gemini 3 Flash achieving the highest accuracy, followed by GPT-5 mini and DeepSeek Chat 3.2, while intra-model variability remained low across all conditions. By question format, the models achieved the highest accuracy on single-answer multiple-choice items, whereas multi-select and True/False questions were more error-prone. By topic, performance was more consistent in normatively explicit areas such as Artifacts, Empiricism, and Product Value, but more fragile in Scrum Values, Self-Managing Teams, and Stakeholders \& Customers. The qualitative analysis showed that errors were systematic rather than random, involving overgeneralization, restrictive wording, compound distractors, and conflicts between common market interpretations and strict Scrum definitions.

Problem

• Large Language Models
• Scrum Certification
• Accuracy
• Stability
• Error Patterns

Innovation

• Large Language Models
• Scrum certification
• prompting strategies
• error pattern analysis
• model stability