Active Nonparametric Two-Sample Testing by Betting on Heterogeneous Data Sources

2025-12-26

AI Summary

This paper addresses the active nonparametric sequential two-sample test problem under multi-source heterogeneous data, aiming to determine whether paired samples originate from the same unknown distribution with minimal expected sampling cost. We propose the first method integrating active learning with the “testing-by-betting” framework: a nonnegative martingale-based betting wealth process is designed; an empirically measurable distinguishability criterion guides source selection; and a vanishing greedy exploration mechanism balances exploration and exploitation. Under no distributional assumptions, the method strictly controls Type-I error at level α and achieves power-one under the alternative. A theoretical upper bound on the expected stopping time is derived, substantially improving upon passive sampling baselines. The core contribution is the first statistically rigorous and sample-efficient active sequential testing paradigm for multi-source nonparametric two-sample testing.

Technology Category

• Machine Learning: Active Learning
• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

We study the problem of active nonparametric sequential two-sample testing over multiple heterogeneous data sources. In each time slot, a decision-maker adaptively selects one of $K$ data sources and receives a paired sample generated from that source for testing. The goal is to decide as quickly as possible whether the pairs are generated from the same distribution or not. The gain achieved by such adaptive sampling (in terms of smaller expected stopping time or larger error exponents) has been well-characterized for parametric models via Chernoff's adaptive MLE selection rule [1]. However, analogous results are not known for the case of nonparametric problems, such as two-sample testing, where we place no restrictions on the distributions. Our main contribution is a general active nonparametric testing procedure that combines an adaptive source-selecting strategy within the testing-by-betting framework of [2] that works under minimal distributional assumptions. In each time slot, our scheme proceeds by selecting a source according to a probability that mixes exploitation, favoring sources with the largest empirical distinguishability, and exploration via a vanishing greedy strategy. The (paired) observations so collected are then used to update the "betting-wealth process", which is a stochastic process guaranteed to be a nonnegative martingale under the null. The procedure stops and rejects the null when the wealth process exceeds an appropriate threshold; an event that is unlikely under the null. We show that our test controls the type-I error at a prespecified level-$α$ under the null, and establish its power-one property and a bound on its expected sample size under the alternative. Our results provide a precise characterization of the improvements achievable by a principled adaptive sampling strategy over its passive analog.

Problem

• Active nonparametric sequential two-sample testing over heterogeneous data sources
• Adaptively selects data sources to decide distribution equality quickly
• General procedure combining adaptive source selection with betting framework

Innovation

• Adaptive source selection with exploitation-exploration strategy
• Nonparametric testing via betting-wealth martingale process
• Sequential two-sample testing with minimal distributional assumptions