PPI-SVRG: Unifying Prediction-Powered Inference and Variance Reduction for Semi-Supervised Optimization

2026-01-29

AI Summary

This work proposes the PPI-SVRG framework for semi-supervised learning scenarios where labels are scarce but predictions from a pretrained model are available. By unifying prediction-based pseudo-inference (PPI) with stochastic variance-reduced gradient (SVRG), the method incorporates pretrained predictions as control variates into the reference gradient, effectively reducing the variance of stochastic optimization. Theoretical analysis reveals a mathematical equivalence between PPI and SVRG, showing that the convergence rate of the proposed algorithm depends solely on the geometric structure of the loss function, while the quality of predictions only affects the size of the convergence neighborhood. Empirical results demonstrate a 43–52% reduction in mean squared error on mean estimation tasks and a 2.7–2.9 percentage point improvement in test accuracy on MNIST with only 10% labeled data.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We study semi-supervised stochastic optimization when labeled data is scarce but predictions from pre-trained models are available. PPI and SVRG both reduce variance through control variates -- PPI uses predictions, SVRG uses reference gradients. We show they are mathematically equivalent and develop PPI-SVRG, which combines both. Our convergence bound decomposes into the standard SVRG rate plus an error floor from prediction uncertainty. The rate depends only on loss geometry; predictions affect only the neighborhood size. When predictions are perfect, we recover SVRG exactly. When predictions degrade, convergence remains stable but reaches a larger neighborhood. Experiments confirm the theory: PPI-SVRG reduces MSE by 43--52\% under label scarcity on mean estimation benchmarks and improves test accuracy by 2.7--2.9 percentage points on MNIST with only 10\% labeled data.

Problem

• semi-supervised optimization
• prediction-powered inference
• variance reduction
• stochastic optimization
• label scarcity

Innovation

• PPI-SVRG
• variance reduction
• semi-supervised optimization
• control variates
• prediction-powered inference