Coupled Training with Privileged Information and Unlabeled Data

2026-05-22

AI Summary

This work addresses the challenge that privileged information—available during training but inaccessible at deployment—can mislead models when it is noisy or weakly informative. To mitigate this issue, the authors propose a joint training framework that simultaneously optimizes a teacher model leveraging privileged information and a student model restricted to inputs available at test time. Through an innovative coupling mechanism and an alternating optimization algorithm, the student selectively distills useful knowledge from the teacher while avoiding the propagation of its errors. Theoretical analysis establishes conditions under which this joint training improves accuracy and supports efficient implementation even for high-dimensional, large-scale models. Experiments on both synthetic and real-world datasets demonstrate that the proposed method significantly outperforms conventional two-stage baselines and exhibits robustness to low-quality privileged information.

Abstract

In many prediction problems, we have extra information during training (for example, measurements that are expensive or slow to collect) that will not be available when the model is deployed. A common strategy is to first train a model that uses all training information, then use its predictions on unlabeled examples to train a second model that only uses the inputs available at test time. However, when the extra training-only information is weak or noisy, this Two-Stage approach can mislead the deployment model and even hurt accuracy. We propose a joint training method that learns the two models together, so the deployment model can benefit from the extra information only when it actually helps, instead of inheriting its mistakes. We provide guarantees that describe when joint training improves prediction accuracy and analyze a simple alternating training algorithm for large, high-dimensional models. Experiments on synthetic data and real-world prediction tasks show that our approach avoids these failures and robustly outperforms standard Two-Stage baselines.

Problem

• privileged information
• unlabeled data
• two-stage training
• prediction accuracy
• model deployment

Innovation

• privileged information
• joint training
• unlabeled data
• two-stage learning
• alternating optimization