SADA: Safe and Adaptive Inference with Multiple Black-Box Predictions

2025-09-25

AI Summary

In real-world settings where labeled data are scarce but unlabeled data abundant, and multiple black-box predictive models (e.g., deep learning, large language models, generative AI) exhibit unknown and heterogeneous quality, reliable aggregation remains challenging. Method: We propose a safe, adaptive multi-predictor aggregation framework that requires no assumptions on model accuracy or underlying data distributions. It is the first method to provably fuse arbitrarily many black-box predictors while guaranteeing that the aggregated estimator performs at least as well as the benchmark trained solely on labeled data. Moreover, it automatically achieves optimal convergence rates—or even semiparametric efficiency—whenever any constituent model attains perfect prediction. The approach integrates ensemble learning, statistical inference, and adaptive algorithm design. Results: Extensive experiments on synthetic and benchmark datasets demonstrate substantial improvements in both inference accuracy and robustness over existing baselines.

Technology Category

• Machine Learning: Ensemble Methods
• Multiagent Systems: Adversarial Agents
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Real-world applications often face scarce labeled data due to the high cost and time requirements of gold-standard experiments, whereas unlabeled data are typically abundant. With the growing adoption of machine learning techniques, it has become increasingly feasible to generate multiple predicted labels using a variety of models and algorithms, including deep learning, large language models, and generative AI. In this paper, we propose a novel approach that safely and adaptively aggregates multiple black-box predictions with unknown quality while preserving valid statistical inference. Our method provides two key guarantees: (i) it never performs worse than using the labeled data alone, regardless of the quality of the predictions; and (ii) if any one of the predictions (without knowing which one) perfectly fits the ground truth, the algorithm adaptively exploits this to achieve either a faster convergence rate or the semiparametric efficiency bound. We demonstrate the effectiveness of the proposed algorithm through experiments on both synthetic and benchmark datasets.

Problem

• Aggregating multiple black-box predictions safely
• Preserving valid statistical inference with unknown quality
• Ensuring performance never worse than labeled data alone

Innovation

• Safely aggregates multiple black-box predictions adaptively
• Never performs worse than using labeled data alone
• Achieves faster convergence if any prediction fits perfectly