On the Optimality of Tracking Fisher Information in Adaptive Testing with Stochastic Binary Responses

2025-10-09

AI Summary

This paper addresses adaptive estimation of a continuous ability parameter from sequential binary responses, aiming to minimize the number of queries while ensuring bounded estimation error. We propose a Fisher information-maximizing active questioning strategy that dynamically selects item difficulty, coupled with a moment-based estimator and a novel stopping statistic derived from Ville’s inequality and large deviations theory to enable real-time convergence assessment. For the first time, we rigorously establish that our method achieves asymptotically optimal sample complexity under both fixed-confidence and fixed-budget settings—overcoming the fundamental theoretical bottleneck in classical approaches, where estimator bias and query distribution are mutually dependent. Our results establish a new paradigm for adaptive assessment that simultaneously guarantees statistical rigor and computational efficiency.

Technology Category

• Machine Learning: Active Learning
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

We study the problem of estimating a continuous ability parameter from sequential binary responses by actively asking questions with varying difficulties, a setting that arises naturally in adaptive testing and online preference learning. Our goal is to certify that the estimate lies within a desired margin of error, using as few queries as possible. We propose a simple algorithm that adaptively selects questions to maximize Fisher information and updates the estimate using a method-of-moments approach, paired with a novel test statistic to decide when the estimate is accurate enough. We prove that this Fisher-tracking strategy achieves optimal performance in both fixed-confidence and fixed-budget regimes, which are commonly invested in the best-arm identification literature. Our analysis overcomes a key technical challenge in the fixed-budget setting -- handling the dependence between the evolving estimate and the query distribution -- by exploiting a structural symmetry in the model and combining large deviation tools with Ville's inequality. Our results provide rigorous theoretical support for simple and efficient adaptive testing procedures.

Problem

• Estimating continuous ability parameters from sequential binary responses
• Minimizing queries while ensuring estimation accuracy within desired margin
• Achieving optimal performance in both fixed-confidence and fixed-budget regimes

Innovation

• Adaptively selects questions to maximize Fisher information
• Updates estimate using method-of-moments approach
• Uses novel test statistic to determine estimation accuracy