Quantifying the Value of Privileged Information Using a PAC-Bayesian Approach

2026-09-11

AI Summary

本文使用PAC-Bayes框架量化特权信息的价值，通过引入一种算法无关的信息论方法，解决了如何衡量特权信息在训练时对模型性能提升的潜力问题。

Abstract

In practice, various learning scenarios provide access to auxiliary features exclusively during training. Incorporating such data to enhance model performance gave rise to a paradigm known as Learning Using Privileged Information (LUPI). While this extra information is intended to improve the resulting model, establishing a generalized, cohesive understanding of how privileged information (PI) transfers useful knowledge remains a challenge. Vapnik's original theory and subsequent works offer performance guarantees in certain cases, but these results are inherently per-algorithm and rely on setting-specific proof approaches. Consequently, a more general framework explaining how and when PI transfers useful knowledge is still missing. To bridge this gap, we introduce an algorithm-agnostic, information-theoretic approach based on the PAC-Bayes framework. Rather than asking whether a particular algorithm exploits PI, we ask how much value it could offer: comparing the tightest achievable risk bound with and without PI yields its potential - an upper limit on the extractable gain. We introduce a metric that quantifies this potential directly from empirical training risk, bypassing the need for test-time data access, and validate our findings in both supervised and unsupervised settings. The results demonstrate a robust correspondence between our training-time metric and true test-time performance gains. Ultimately, this work takes a necessary step toward an information-theoretic understanding of LUPI, and quantifying the potential of privileged features before committing to a model.

Problem

• Privileged Information
• PAC-Bayesian
• Algorithm-agnostic
• Information-theoretic
• Performance Guarantee

Innovation

• PAC-Bayes
• privileged information
• algorithm-agnostic
• information-theoretic
• performance gain