The Good, the Bad, and the Sampled: a No-Regret Approach to Safe Online Classification

2025-10-01

AI Summary

This paper addresses cost-minimization in online binary disease diagnosis: minimizing expensive diagnostic tests while guaranteeing, with probability ≥1−δ, that the misclassification rate remains within a given tolerance α—under an unknown underlying logical model. We propose the first safe, no-regret sequential algorithm satisfying the α-misclassification constraint. Our method dynamically interleaves label acquisition with contextual distribution estimation to adaptively update a conservative risk threshold. It integrates online logistic regression, data-driven threshold generation, and tight probabilistic bound analysis. Theoretically, our algorithm incurs test cost only O(√T) higher than the oracle optimal solution under known models, while strictly enforcing the α-misclassification constraint. Extensive simulations validate both its safety (i.e., guaranteed constraint satisfaction) and efficiency (i.e., significantly reduced testing cost compared to baselines).

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We study the problem of sequentially testing individuals for a binary disease outcome whose true risk is governed by an unknown logistic model. At each round, a patient arrives with feature vector $x_t$, and the decision maker may either pay to administer a (noiseless) diagnostic test--revealing the true label--or skip testing and predict the patient's disease status based on their feature vector and prior history. Our goal is to minimize the total number of costly tests required while guaranteeing that the fraction of misclassifications does not exceed a prespecified error tolerance $alpha$, with probability at least $1-delta$. To address this, we develop a novel algorithm that interleaves label-collection and distribution estimation to estimate both $ heta^{*}$ and the context distribution $P$, and computes a conservative, data-driven threshold $ au_t$ on the logistic score $|x_t^ op heta|$ to decide when testing is necessary. We prove that, with probability at least $1-delta$, our procedure does not exceed the target misclassification rate, and requires only $O(sqrt{T})$ excess tests compared to the oracle baseline that knows both $ heta^{*}$ and the patient feature distribution $P$. This establishes the first no-regret guarantees for error-constrained logistic testing, with direct applications to cost-sensitive medical screening. Simulations corroborate our theoretical guarantees, showing that in practice our procedure efficiently estimates $ heta^{*}$ while retaining safety guarantees, and does not require too many excess tests.

Problem

• Sequentially testing individuals for binary disease with unknown logistic model
• Minimizing costly diagnostic tests while controlling misclassification rate
• Developing no-regret algorithm for safe online classification with error constraints

Innovation

• Interleaves label-collection and distribution estimation
• Computes data-driven threshold for testing necessity
• Ensures error rate constraint with no-regret guarantees