The Role of Hyperparameters in Predictive Multiplicity

2025-03-13

AI Summary

This paper addresses prediction multiplicity—a critical issue in machine learning wherein distinct models trained on identical data yield divergent predictions for the same input, undermining reliability in high-stakes domains such as credit scoring and clinical diagnosis. We systematically evaluate six model families—Elastic Net, decision trees, k-NN, SVM, random forests, and XGBoost—across 21 tabular datasets. Using exhaustive hyperparameter grid search and rigorous analysis of prediction disagreement distributions, we quantitatively characterize the trade-off between predictive performance gains and increased prediction inconsistency induced by key hyperparameters (e.g., λ, γ, α). Results demonstrate that hyperparameter optimization consistently improves accuracy but exacerbates prediction multiplicity; XGBoost exhibits the highest prediction instability. Crucially, prediction multiplicity presents a double-edged sword: it may serve as an avenue for fairness-aware model selection yet simultaneously introduces arbitrariness into consequential decisions. This work provides the first empirical quantification of this fundamental trade-off, offering actionable insights for trustworthy ML deployment.

Technology Category

• Machine Learning: Ensemble Methods
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This paper investigates the critical role of hyperparameters in predictive multiplicity, where different machine learning models trained on the same dataset yield divergent predictions for identical inputs. These inconsistencies can seriously impact high-stakes decisions such as credit assessments, hiring, and medical diagnoses. Focusing on six widely used models for tabular data - Elastic Net, Decision Tree, k-Nearest Neighbor, Support Vector Machine, Random Forests, and Extreme Gradient Boosting - we explore how hyperparameter tuning influences predictive multiplicity, as expressed by the distribution of prediction discrepancies across benchmark datasets. Key hyperparameters such as lambda in Elastic Net, gamma in Support Vector Machines, and alpha in Extreme Gradient Boosting play a crucial role in shaping predictive multiplicity, often compromising the stability of predictions within specific algorithms. Our experiments on 21 benchmark datasets reveal that tuning these hyperparameters leads to notable performance improvements but also increases prediction discrepancies, with Extreme Gradient Boosting exhibiting the highest discrepancy and substantial prediction instability. This highlights the trade-off between performance optimization and prediction consistency, raising concerns about the risk of arbitrary predictions. These findings provide insight into how hyperparameter optimization leads to predictive multiplicity. While predictive multiplicity allows prioritizing domain-specific objectives such as fairness and reduces reliance on a single model, it also complicates decision-making, potentially leading to arbitrary or unjustified outcomes.

Problem

• Investigates hyperparameters' impact on predictive multiplicity.
• Explores trade-off between performance and prediction consistency.
• Highlights risks of arbitrary predictions in high-stakes decisions.

Innovation

• Hyperparameter tuning impacts predictive multiplicity.
• Key hyperparameters influence prediction discrepancies.
• Trade-off between performance and prediction stability.