Mining the Gold: Student-AI Chat Logs as Rich Sources for Automated Knowledge Gap Detection

2025-12-26

AI Summary

In large-enrollment computer science courses, instructors struggle to promptly identify students’ spontaneous knowledge gaps. To address this, we propose QueryQuilt—the first multi-agent LLM framework that automatically mines collective knowledge gaps from student-initiated AI tutoring dialogues. Our method leverages students’ self-generated questions as natural probes for latent knowledge deficits, moving beyond instructor-driven interactions. It employs a dual-agent architecture: a dialogue agent conducts progressive, Socratic-style probing to surface implicit misconceptions, while a knowledge-gap identification agent performs cross-conversation clustering and pattern modeling. Evaluated on synthetic data, QueryQuilt achieves 100% gap detection accuracy; on real student dialogue logs, it attains 95% coverage of empirically validated gaps. Furthermore, it generates interpretable, class-level knowledge-gap heatmaps—enabling targeted, data-informed instructional interventions.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Question Answering
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

With the significant increase in enrollment in computing-related programs over the past 20 years, lecture sizes have grown correspondingly. In large lectures, instructors face challenges on identifying students' knowledge gaps timely, which is critical for effective teaching. Existing classroom response systems rely on instructor-initiated interactions, which limits their ability to capture the spontaneous knowledge gaps that naturally emerge during lectures. With the widespread adoption of LLMs among students, we recognize these student-AI dialogues as a valuable, student-centered data source for identifying knowledge gaps. In this idea paper, we propose QueryQuilt, a multi-agent LLM framework that automatically detects common knowledge gaps in large-scale lectures by analyzing students' chat logs with AI assistants. QueryQuilt consists of two key components: (1) a Dialogue Agent that responds to student questions while employing probing questions to reveal underlying knowledge gaps, and (2) a Knowledge Gap Identification Agent that systematically analyzes these dialogues to identify knowledge gaps across the student population. By generating frequency distributions of identified gaps, instructors can gain comprehensive insights into class-wide understanding. Our evaluation demonstrates promising results, with QueryQuilt achieving 100% accuracy in identifying knowledge gaps among simulated students and 95% completeness when tested on real student-AI dialogue data. These initial findings indicate the system's potential for facilitate teaching in authentic learning environments. We plan to deploy QueryQuilt in actual classroom settings for comprehensive evaluation, measuring its detection accuracy and impact on instruction.

Problem

• Automatically detects knowledge gaps in large lectures
• Analyzes student-AI chat logs to identify common misunderstandings
• Provides instructors with insights into class-wide comprehension

Innovation

• Multi-agent LLM framework analyzes student-AI chat logs
• Dialogue Agent uses probing questions to reveal knowledge gaps
• Knowledge Gap Identification Agent detects common gaps across students