Conformal Prediction with Macro-Coverage Guarantees

2026-06-26

AI Summary

This work addresses the challenge in long-tailed classification where conventional conformal prediction struggles to balance coverage quality across classes: marginal coverage favors frequent classes, while class-conditional coverage suffers from insufficient calibration samples for rare classes. The authors propose a label-weighted conformal prediction method that achieves macro-coverage guarantees under limited sample sizes and generalizes to weighted average coverage objectives over arbitrary class groupings. They provide the first finite-sample theoretical guarantees for both macro-coverage and its generalized form, characterize the minimal prediction sets satisfying these targets, and design corresponding conformal scoring functions. Experiments on two large-scale image classification benchmarks demonstrate that the proposed approach significantly improves coverage equity and reliability across all classes in long-tailed settings.

Abstract

Prediction sets should have high coverage to be useful, but some coverage notions are more practically relevant than others. In the classification setting, class-conditional coverage requires that the prediction set (i.e., the set of candidate labels for a new test point) must achieve the target accuracy level within each class, which may be challenging to satisfy when many classes are rare and have few calibration points. At the other extreme, marginal coverage requires only that coverage holds on average over the distribution of all classes, which can lead to low-probability labels being essentially ignored. To find a middle ground, recent work has introduced macro-coverage, defined as the unweighted average of class-conditional coverages. Macro-coverage offers a compromise between marginal coverage and class-conditional coverage that is particularly appropriate for long-tailed settings. In this work, we show that label-weighted conformal prediction can be used to produce prediction sets with a finite-sample macro-coverage guarantee, and more generally a guarantee on a family of generalized macro-coverage objectives that aggregate coverage at the level of arbitrary class groupings and take a weighted average. We further characterize the form of the smallest prediction sets satisfying a given generalized macro-coverage objective and propose a corresponding conformal score function. We validate our theoretical results on two large-scale image classification datasets.

Problem

• conformal prediction
• macro-coverage
• class-conditional coverage
• marginal coverage
• long-tailed classification

Innovation

• Conformal Prediction
• Macro-Coverage
• Class-Conditional Coverage
• Label-Weighted Calibration
• Prediction Sets