H2ST: Hierarchical Two-Sample Tests for Continual Out-of-Distribution Detection

2025-03-19

AI Summary

Real-time out-of-distribution (OOD) detection for unlabeled streaming data in open-world task-incremental learning (TIL) remains challenging due to reliance on heuristic thresholds and model output confidence. Method: We propose the first threshold-free, model-output-agnostic hierarchical two-sample testing framework. It leverages nonparametric statistical tests—specifically Maximum Mean Discrepancy (MMD) and Kullback–Leibler (KL) divergence—in feature space to construct an online hierarchical distribution modeling and task-level hypothesis testing architecture, enabling joint OOD discrimination and task attribution. Contribution/Results: Unlike state-of-the-art methods, our approach decouples OOD detection from model performance assumptions. Evaluated on multiple open-world TIL benchmarks, it achieves significantly higher OOD detection rates and lower false positive rates. Moreover, it delivers interpretable, task-level identification results while maintaining high computational efficiency and detection accuracy.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Task Incremental Learning (TIL) is a specialized form of Continual Learning (CL) in which a model incrementally learns from non-stationary data streams. Existing TIL methodologies operate under the closed-world assumption, presuming that incoming data remains in-distribution (ID). However, in an open-world setting, incoming samples may originate from out-of-distribution (OOD) sources, with their task identities inherently unknown. Continually detecting OOD samples presents several challenges for current OOD detection methods: reliance on model outputs leads to excessive dependence on model performance, selecting suitable thresholds is difficult, hindering real-world deployment, and binary ID/OOD classification fails to provide task-level identification. To address these issues, we propose a novel continual OOD detection method called the Hierarchical Two-sample Tests (H2ST). H2ST eliminates the need for threshold selection through hypothesis testing and utilizes feature maps to better exploit model capabilities without excessive dependence on model performance. The proposed hierarchical architecture enables task-level detection with superior performance and lower overhead compared to non-hierarchical classifier two-sample tests. Extensive experiments and analysis validate the effectiveness of H2ST in open-world TIL scenarios and its superiority to the existing methods. Code is available at href{https://github.com/YuhangLiuu/H2ST}{https://github.com/YuhangLiuu/H2ST}.

Problem

• Detects out-of-distribution samples in continual learning
• Eliminates threshold selection through hypothesis testing
• Enables task-level detection with hierarchical architecture

Innovation

• Hierarchical Two-sample Tests for OOD detection
• Eliminates threshold selection via hypothesis testing
• Uses feature maps to enhance model capabilities