AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models

2026-06-19

AI Summary

This study addresses the limitation of existing computerized adaptive testing (CAT) research, which predominantly relies on static offline data and fails to authentically simulate dynamic assessment processes. To overcome this, the work proposes a high-fidelity end-to-end simulation framework powered by multi-agent large language models, comprising three collaborative agents: examinee, item selection, and supervision. The examinee agent integrates memory retrieval with chain-of-thought reasoning, while the item selection strategy combines coarse-to-fine bucketing with knowledge graph exploration. The supervision module employs a dual-audit mechanism and robust updating protocols. Experimental results demonstrate that the system accurately estimates examinee proficiency on real-world data, selects items that balance difficulty adaptation with instructional coherence, exhibits strong robustness under data sparsity, and aligns its interaction logic with human pedagogical intuition.

Abstract

Computerized Adaptive Testing (CAT), as a key technology for personalized education, aims to accurately assess examinee proficiency by retrieving exercises dynamically matching current ability estimates. However, existing CAT research is constrained by limitations of static offline data and isolated component optimization. Restricted by partial labels in offline logs, researchers degrade the dynamic assessment process into static sequence prediction. Current research focuses on isolated perspectives, e.g., selection or diagnosis, neglecting the overall CAT interaction process. To address this, we propose AgentCAT, a Large Language Model-based multi-agent simulation system, to construct a high-fidelity benchmarking environment for dynamic testing. This framework comprises three modules: (1) The examinee agent with memory retrieval and Chain-of-Thought reasoning simulates responses based on cognitive profiles; (2) The selection agent uses coarse-to-fine bucketing and knowledge graph exploration to balance local difficulty and global coverage; (3) The supervisor uses dual-auditing and robust update to ensure convergence and validity. To validate the framework, we evaluated on two real-world datasets across three dimensions: macro-level ability convergence, micro-level interaction logic, and data sparsity resilience. Results show AgentCAT achieves effective ability estimation, and its selection strategy balances difficulty adaptation and instructional coherence, aligning with human pedagogical intuition.

Problem

• Computerized Adaptive Testing
• dynamic assessment
• offline data limitation
• isolated component optimization
• adaptive testing simulation

Innovation

• Computerized Adaptive Testing
• Multi-Agent System
• Large Language Models
• Knowledge Graph
• Chain-of-Thought Reasoning