Weighted Aggregation of Conformity Scores for Classification

2024-07-14

arXiv.org

AI Summary

Conformal prediction for multi-class classification often suffers from inefficiency and overly large prediction sets due to reliance on a single scoring function. To address this, we propose a weighted ensemble of multiple scoring functions within the conformal prediction framework. Our method learns data-driven weights via joint optimization grounded in empirical risk minimization, integrating Vapnik–Chervonenkis (VC) theory with convex optimization. Crucially, we establish, for the first time, a theoretical connection between weighted score aggregation and VC subgraph classes—thereby enabling provably optimal multi-score fusion. Under strict coverage guarantees (e.g., 90%), our approach significantly reduces prediction set size, achieving an average reduction of 12.6% across multiple benchmark datasets. It consistently outperforms state-of-the-art single-score conformal methods in both efficiency and predictive performance.

Technology Category

• Machine Learning: Ensemble Methods
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Conformal prediction is a powerful framework for constructing prediction sets with valid coverage guarantees in multi-class classification. However, existing methods often rely on a single score function, which can limit their efficiency and informativeness. We propose a novel approach that combines multiple score functions to improve the performance of conformal predictors by identifying optimal weights that minimize prediction set size. Our theoretical analysis establishes a connection between the weighted score functions and subgraph classes of functions studied in Vapnik-Chervonenkis theory, providing a rigorous mathematical basis for understanding the effectiveness of the proposed method. Experiments demonstrate that our approach consistently outperforms single-score conformal predictors while maintaining valid coverage, offering a principled and data-driven way to enhance the efficiency and practicality of conformal prediction in classification tasks.

Problem

• Improves conformal prediction by combining multiple score functions.
• Identifies optimal weights to minimize prediction set size.
• Enhances efficiency and practicality in classification tasks.

Innovation

• Combines multiple score functions for classification
• Uses optimal weights to minimize prediction set size
• Connects weighted scores to Vapnik-Chervonenkis theory