What is the Right Notion of Distance between Predict-then-Optimize Tasks?

2024-09-11

arXiv.org

AI Summary

In the Predict-then-Optimize (PtO) paradigm, conventional dataset distances—based on feature or label distribution divergence—fail to reflect model transfer performance on downstream decision tasks, as they ignore decision regret, the core evaluation metric. To address this, we propose the first **decision-aware dataset distance**, which explicitly incorporates the downstream optimization task into distance modeling. We theoretically establish its tight connection to PtO adaptation error and derive an interpretable, regret-based adaptation error bound. Empirical evaluation across three canonical PtO tasks demonstrates that our distance accurately predicts cross-dataset transfer performance, significantly outperforming traditional metrics grounded in prediction error or distributional divergence. This work introduces a novel paradigm for dataset similarity assessment and model transfer in PtO settings.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Comparing datasets is a fundamental task in machine learning, essential for various learning paradigms; from evaluating train and test datasets for model generalization to using dataset similarity for detecting data drift. While traditional notions of dataset distances offer principled measures of similarity, their utility has largely been assessed through prediction error minimization. However, in Predict-then-Optimize (PtO) frameworks, where predictions serve as inputs for downstream optimization tasks, model performance is measured through decision regret minimization rather than prediction error minimization. In this work, we (i) show that traditional dataset distances, which rely solely on feature and label dimensions, lack informativeness in the PtO context, and (ii) propose a new dataset distance that incorporates the impacts of downstream decisions. Our results show that this decision-aware dataset distance effectively captures adaptation success in PtO contexts, providing a PtO adaptation bound in terms of dataset distance. Empirically, we show that our proposed distance measure accurately predicts transferability across three different PtO tasks from the literature.

Problem

• Measure dataset distance considering downstream optimization decisions
• Address limitations of traditional feature-label distances in PtO tasks
• Propose OTD3 for better adaptation success prediction in PtO

Innovation

• Optimal Transport Decision-aware Dataset Distance
• Incorporates downstream decisions in dataset distance
• Effectively captures adaptation success in PtO