DANCE: Doubly Adaptive Neighborhood Conformal Estimation

2026-02-24

AI Summary

Pretrained deep models often yield poorly calibrated predictions in classification tasks, leading to overly large and inefficient conformal prediction sets when using conventional logit-based approaches. To address this, this work proposes a dual local adaptive nearest-neighbor conformal prediction method grounded in embedding representations. By modeling task-adaptive kernel regression within the learned embedding space and introducing two novel nonconformity scoring mechanisms, the method achieves dual adaptation to local data structure. The resulting framework provides end-to-end uncertainty quantification and consistently outperforms existing local, task-adaptive, and zero-shot conformal baselines across multiple datasets. It maintains strict statistical validity while substantially reducing prediction set sizes, thereby enhancing both compactness and generalization.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

The recent developments of complex deep learning models have led to unprecedented ability to accurately predict across multiple data representation types. Conformal prediction for uncertainty quantification of these models has risen in popularity, providing adaptive, statistically-valid prediction sets. For classification tasks, conformal methods have typically focused on utilizing logit scores. For pre-trained models, however, this can result in inefficient, overly conservative set sizes when not calibrated towards the target task. We propose DANCE, a doubly locally adaptive nearest-neighbor based conformal algorithm combining two novel nonconformity scores directly using the data's embedded representation. DANCE first fits a task-adaptive kernel regression model from the embedding layer before using the learned kernel space to produce the final prediction sets for uncertainty quantification. We test against state-of-the-art local, task-adapted and zero-shot conformal baselines, demonstrating DANCE's superior blend of set size efficiency and robustness across various datasets.

Problem

• conformal prediction
• uncertainty quantification
• pre-trained models
• prediction sets
• classification

Innovation

• conformal prediction
• adaptive uncertainty quantification
• nearest-neighbor
• embedding representation
• task-adaptive kernel