Reliable algorithm selection for machine learning-guided design

2025-03-26

AI Summary

In machine learning–guided design, a critical challenge is reliably selecting a design algorithm that satisfies user-specified success criteria—e.g., ensuring ≥10% of generated designs exceed a threshold on a target property. Method: We propose the first prediction-powered inference framework for algorithm selection. It integrates predictions from a learned model with held-out labeled data, employing density-ratio weighting for calibration. The method provides theoretical guarantees that the selected algorithm satisfies the desired success probability constraint with high confidence, supporting both known and estimable density ratios; it either returns the optimal algorithm or certifies infeasibility. Crucially, it jointly optimizes the predictive and generative models. Results: Evaluated on simulated protein and RNA design tasks, our approach significantly improves accuracy in identifying successful algorithms while strictly enforcing user-specified probabilistic constraints—empirically validating the practicality of its theoretical guarantees.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Algorithms for machine learning-guided design, or design algorithms, use machine learning-based predictions to propose novel objects with desired property values. Given a new design task -- for example, to design novel proteins with high binding affinity to a therapeutic target -- one must choose a design algorithm and specify any hyperparameters and predictive and/or generative models involved. How can these decisions be made such that the resulting designs are successful? This paper proposes a method for design algorithm selection, which aims to select design algorithms that will produce a distribution of design labels satisfying a user-specified success criterion -- for example, that at least ten percent of designs' labels exceed a threshold. It does so by combining designs' predicted property values with held-out labeled data to reliably forecast characteristics of the label distributions produced by different design algorithms, building upon techniques from prediction-powered inference. The method is guaranteed with high probability to return design algorithms that yield successful label distributions (or the null set if none exist), if the density ratios between the design and labeled data distributions are known. We demonstrate the method's effectiveness in simulated protein and RNA design tasks, in settings with either known or estimated density ratios.

Problem

• Selecting reliable algorithms for machine learning-guided design tasks
• Ensuring design algorithms meet user-specified success criteria
• Forecasting label distributions for novel protein and RNA designs

Innovation

• Combines predicted values with held-out data
• Forecasts label distributions reliably
• Guarantees successful algorithm selection