Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models

2025-12-10

AI Summary

Overreliance on large language models (LLMs) for pedagogical question generation in learning analytics suffers from inefficiency, opacity, and misalignment with instructional objectives. Method: This paper proposes a two-stage “generate-verify” framework leveraging small language models (SLMs). In the first stage, an SLM generates diverse candidate questions; in the second, probabilistic verification and re-ranking—guided by structured reasoning—select questions exhibiting high answer definiteness and strong pedagogical alignment. Contribution/Results: To our knowledge, this is the first work to deeply integrate SLM-based text generation with probabilistic inference for educational question generation, thereby extending SLMs’ capabilities in complex instructional tasks. Evaluated via dual human–machine assessment (seven domain experts + LLM-based evaluation), the method achieves LLM-level performance in answer clarity and learning objective consistency, demonstrating that lightweight models—when embedded in a carefully designed architecture—can deliver high-fidelity, educationally grounded question generation.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We explore the use of small language models (SLMs) for automatic question generation as a complement to the prevalent use of their large counterparts in learning analytics research. We present a novel question generation pipeline that leverages both the text generation and the probabilistic reasoning abilities of SLMs to generate high-quality questions. Adopting a "generate-then-validate" strategy, our pipeline first performs expansive generation to create an abundance of candidate questions and refine them through selective validation based on novel probabilistic reasoning. We conducted two evaluation studies, one with seven human experts and the other with a large language model (LLM), to assess the quality of the generated questions. Most judges (humans or LLMs) agreed that the generated questions had clear answers and generally aligned well with the intended learning objectives. Our findings suggest that an SLM can effectively generate high-quality questions when guided by a well-designed pipeline that leverages its strengths.

Problem

• Develop a question generation pipeline using small language models
• Generate and validate questions through probabilistic reasoning
• Assess question quality via human experts and large language models

Innovation

• SLMs generate candidate questions expansively
• Validate questions via probabilistic reasoning
• Pipeline combines generation and validation steps