Agentic Knowledge Tracing: A Multi-Agent LLM Architecture for Stealth Assessment of Financial Literacy in Serious Games

2026-06-23

AI Summary

This study proposes a novel, unobtrusive method for assessing players’ financial literacy without disrupting their learning experience in serious games. By integrating structured behavioral logs, domain-specific agents, Bayesian Knowledge Tracing (BKT), and an expert adjudication mechanism into a multi-agent large language model architecture—termed Agentic BKT—the approach uniquely combines domain decomposition with conversational-level reasoning to implicitly and accurately model four key dimensions: risk, investment, consumption, and credit. Evaluated across 193 K–12 students and 264 gameplay sessions, the estimated mastery scores demonstrated significant correlations with learning gains (r = 0.276) and post-test performance (r = 0.333). The method achieves predictive validity three times higher than a single-LLM baseline, substantially enhancing the validity and practicality of implicit assessment in educational gaming contexts.

Abstract

Assessing financial literacy during gameplay without disrupting the learning experience remains a key challenge in serious games for education. We present the Agentic BKT pipeline, a multi-agent large language model architecture for stealth assessment of financial competencies from open-ended gameplay events. The pipeline processes events from a 2D platformer serious game aligned with the OECD/INFE financial literacy framework through four phases: (1) the game captures every player decision as a structured event log; (2) an LLM event classifier labels each action on a four-point rubric validated against three domain experts (Fleiss kappa = 0.624, substantial agreement); (3) four domain-specific agents specializing in risk mitigation, investing, spending, and credit management perform session-level reasoning over behavioral trajectories, feeding per-competency Bayesian Knowledge Tracing that estimates mastery within each domain; and (4) an expert judge agent synthesizes the domain-level estimates into an overall mastery score. Evaluated with 193 K-12 participants across 264 game sessions, the Agentic BKT pipeline yields mastery estimates significantly correlated with learning gain (r = 0.276, p = 0.0001) and post-test scores (r = 0.333, p < 0.0001) while showing no correlation with pre-test scores, providing both convergent and discriminant validity. The multi-agent approach approximately triples the predictive validity of a single-LLM baseline (r = 0.095, not significant) in this study, demonstrating that domain decomposition and session-level reasoning play a central role in capturing the multidimensional nature of financial literacy from gameplay

Problem

• stealth assessment
• financial literacy
• serious games
• knowledge tracing
• educational assessment

Innovation

• Agentic Knowledge Tracing
• Multi-Agent LLM
• Stealth Assessment
• Bayesian Knowledge Tracing
• Financial Literacy