Cluster Frequency Conformal Prediction for Local Coverage

2026-05-24

AI Summary

Standard conformal prediction often exhibits insufficient coverage for specific classes or subgroups in multiclass classification, falling short of the reliability demands in high-stakes settings. This work proposes CFCP, a plug-and-play framework that, for the first time, incorporates clustering-based frequency information into conformal prediction. By clustering in the representation space, estimating cluster-level label frequencies, and combining global priors with reliability-aware shrinkage, CFCP constructs localized probability vectors for each sample before applying conformal calibration. This approach adaptively captures local structure, significantly enhancing class-conditional coverage stability while preserving marginal validity. Empirical results demonstrate that CFCP achieves the best class-wise coverage in 15 out of 16 comparisons across image and text benchmarks, with more efficient prediction set sizes.

Abstract

Conformal prediction provides distribution-free coverage guarantees, but in many-class classification it may still under-cover specific classes or subpopulations, preventing safe deployment in high-stakes applications. We propose Cluster Frequency Conformal Prediction (CFCP), a plug-in framework that adapts conformal prediction to local structure in a learned representation space. CFCP clusters learned embeddings, estimates cluster-level label-frequency distributions from calibration data, and for each test point constructs a sample-specific probability vector by softly mixing nearby cluster distributions regularized with global-prior and reliability-aware shrinkage. This vector is then conformalized using standard set constructors. In the disjoint-split regime, CFCP inherits standard finite-sample marginal validity. Under additional assumptions, CFCP further admits a local-validity interpretation. Since representation clusters aggregate locally similar samples, their empirical class frequencies provide a stable estimate of local label ambiguity. Across image and text benchmarks, CFCP achieves the best class coverage in 15/16 dataset/score-family comparisons and a competitive prediction set size efficiency, with several settings substantially more efficient. Overall, our results show that cluster-frequency information provides an effective localized signal for improving classwise reliability in many-class conformal prediction.

Problem

• conformal prediction
• class coverage
• local validity
• many-class classification
• subpopulation under-coverage

Innovation

• Conformal Prediction
• Local Coverage
• Representation Clustering
• Label-Frequency Estimation
• Reliability-Aware Shrinkage