Conformal novelty detection with false discovery rate control at the boundary

2026-01-06

AI Summary

This work addresses the issue that the traditional Benjamini–Hochberg (BH) procedure, when applied in conformal novelty detection, tends to yield an excessive number of false discoveries near decision boundaries, leading to overly optimistic results. For the first time, the authors introduce boundary false discovery rate (bFDR) control into the conformal prediction framework and propose several novel procedures that rigorously guarantee bFDR control. They further demonstrate that the classical support line method may fail under this setting. By integrating conformal prediction, p-value calibration, and multiple hypothesis testing theory with a refined support line correction, the proposed algorithms theoretically ensure valid bFDR control. Extensive experiments on both synthetic and real-world datasets confirm the superiority and practical utility of the approach.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Conformal novelty detection is a classical machine learning task for which uncertainty quantification is essential for providing reliable results. Recent work has shown that the BH procedure applied to conformal p-values controls the false discovery rate (FDR). Unfortunately, the BH procedure can lead to over-optimistic assessments near the rejection threshold, with an increase of false discoveries at the margin as pointed out by Soloff et al. (2024). This issue is solved therein by the support line (SL) correction, which is proven to control the boundary false discovery rate (bFDR) in the independent, non-conformal setting. The present work extends the SL method to the conformal setting: first, we show that the SL procedure can violate the bFDR control in this specific setting. Second, we propose several alternatives that provably control the bFDR in the conformal setting. Finally, numerical experiments with both synthetic and real data support our theoretical findings and show the relevance of the new proposed procedures.

Problem

• conformal novelty detection
• false discovery rate
• boundary false discovery rate
• uncertainty quantification
• p-values

Innovation

• conformal novelty detection
• boundary false discovery rate
• support line correction
• false discovery rate control
• conformal p-values