Offline Model-Based Optimization by Learning to Rank

2024-10-15

arXiv.org

AI Summary

Offline model-based optimization (MBO) faces a fundamental challenge: regression models trained on static datasets suffer from out-of-distribution errors, leading to overestimation of suboptimal designs and misguiding the optimization process. This work observes that MBO’s core objective is to **identify promising design rankings**, not to predict absolute performance scores accurately. Accordingly, we propose the first integration of **Learning to Rank (LTR)** into offline MBO. Instead of minimizing mean squared error, our method employs pairwise or listwise ranking losses within an offline reinforcement learning framework to explicitly model relative design preferences. We further derive a theoretical upper bound on the generalization error of ranking loss in this setting. Evaluated across diverse benchmark tasks, our approach consistently outperforms 20 state-of-the-art methods, achieving superior robustness and higher-quality optimal solutions.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Offline model-based optimization (MBO) aims to identify a design that maximizes a black-box function using only a fixed, pre-collected dataset of designs and their corresponding scores. A common approach in offline MBO is to train a regression-based surrogate model by minimizing mean squared error (MSE) and then find the best design within this surrogate model by different optimizers (e.g., gradient ascent). However, a critical challenge is the risk of out-of-distribution errors, i.e., the surrogate model may typically overestimate the scores and mislead the optimizers into suboptimal regions. Prior works have attempted to address this issue in various ways, such as using regularization techniques and ensemble learning to enhance the robustness of the model, but it still remains. In this paper, we argue that regression models trained with MSE are not well-aligned with the primary goal of offline MBO, which is to select promising designs rather than to predict their scores precisely. Notably, if a surrogate model can maintain the order of candidate designs based on their relative score relationships, it can produce the best designs even without precise predictions. To validate it, we conduct experiments to compare the relationship between the quality of the final designs and MSE, finding that the correlation is really very weak. In contrast, a metric that measures order-maintaining quality shows a significantly stronger correlation. Based on this observation, we propose learning a ranking-based model that leverages learning to rank techniques to prioritize promising designs based on their relative scores. We show that the generalization error on ranking loss can be well bounded. Empirical results across diverse tasks demonstrate the superior performance of our proposed ranking-based models than twenty existing methods.

Problem

• Offline MBO aims to maximize black-box function using fixed dataset.
• Regression models often overestimate scores, leading to suboptimal designs.
• Proposed ranking-based model prioritizes designs by relative scores, improving performance.

Innovation

• Uses ranking-based model for design optimization
• Focuses on relative score relationships, not precise predictions
• Demonstrates superior performance over existing methods