Harnessing the Power of Large Language Models for Software Testing Education: A Focus on ISTQB Syllabus

2025-10-25

AI Summary

This study addresses the integration of large language models (LLMs) into higher education for software testing under the ISTQB certification framework. Methodologically, we construct the first standardized ISTQB examination dataset spanning 11 years and comprising 1,145 questions; design domain-specific prompt engineering strategies; and establish a dedicated evaluation framework jointly assessing knowledge alignment and explanation quality. Contributions include the first systematic validation of mainstream LLMs on ISTQB exam preparation tasks—demonstrating high accuracy and explanatory fidelity across multiple models; proposing a reproducible AI-augmented pedagogical integration framework; and open-sourcing both the dataset and evaluation benchmark. These resources constitute foundational infrastructure and an actionable paradigm for AI-driven software engineering education.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Software testing is a critical component in the software engineering field and is important for software engineering education. Thus, it is vital for academia to continuously improve and update educational methods to reflect the current state of the field. The International Software Testing Qualifications Board (ISTQB) certification framework is globally recognized and widely adopted in industry and academia. However, ISTQB-based learning has been rarely applied with recent generative artificial intelligence advances. Despite the growing capabilities of large language models (LLMs), ISTQB-based learning and instruction with LLMs have not been thoroughly explored. This paper explores and evaluates how LLMs can complement the ISTQB framework for higher education. The findings present four key contributions: (i) the creation of a comprehensive ISTQB-aligned dataset spanning over a decade, consisting of 28 sample exams and 1,145 questions; (ii) the development of a domain-optimized prompt that enhances LLM precision and explanation quality on ISTQB tasks; (iii) a systematic evaluation of state-of-the-art LLMs on this dataset; and (iv) actionable insights and recommendations for integrating LLMs into software testing education. These findings highlight the promise of LLMs in supporting ISTQB certification preparation and offer a foundation for their broader use in software engineering at higher education.

Problem

• Enhancing ISTQB-based learning using large language models
• Developing optimized prompts to improve LLM performance
• Evaluating LLMs' effectiveness for software testing education

Innovation

• Created a decade-spanning ISTQB dataset with 1145 questions
• Developed domain-optimized prompts to enhance LLM precision
• Systematically evaluated state-of-the-art LLMs on ISTQB tasks