Risk-aware black-box portfolio construction using Bayesian optimization with adaptive weighted Lagrangian estimator

2025-04-18

AI Summary

To address the opacity of models, limited observation budgets, and uncontrolled risk in black-box portfolio management for finance, this paper proposes a risk-aware Bayesian optimization framework. Unlike conventional approaches that solely maximize expected return, we introduce an adaptive weighted Lagrangian estimator that jointly optimizes expected return maximization and observation variance minimization within a Gaussian process surrogate modeling framework. Crucially, we are the first to embed a variance constraint directly into the Lagrangian dual formulation and construct a variance-aware acquisition function. Extensive evaluation across five backtesting scenarios and three classes of black-box stock portfolio models demonstrates significant superiority over baseline methods. Ablation studies confirm the framework’s dual efficacy in mitigating risk accumulation and enhancing robust return performance.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Optimization

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Existing portfolio management approaches are often black-box models due to safety and commercial issues in the industry. However, their performance can vary considerably whenever market conditions or internal trading strategies change. Furthermore, evaluating these non-transparent systems is expensive, where certain budgets limit observations of the systems. Therefore, optimizing performance while controlling the potential risk of these financial systems has become a critical challenge. This work presents a novel Bayesian optimization framework to optimize black-box portfolio management models under limited observations. In conventional Bayesian optimization settings, the objective function is to maximize the expectation of performance metrics. However, simply maximizing performance expectations leads to erratic optimization trajectories, which exacerbate risk accumulation in portfolio management. Meanwhile, this can lead to misalignment between the target distribution and the actual distribution of the black-box model. To mitigate this problem, we propose an adaptive weight Lagrangian estimator considering dual objective, which incorporates maximizing model performance and minimizing variance of model observations. Extensive experiments demonstrate the superiority of our approach over five backtest settings with three black-box stock portfolio management models. Ablation studies further verify the effectiveness of the proposed estimator.

Problem

• Optimize black-box portfolio models with limited observations
• Control financial risk while maximizing model performance
• Align target and actual distributions in Bayesian optimization

Innovation

• Bayesian optimization for black-box portfolios
• Adaptive weighted Lagrangian estimator
• Dual objective: maximize performance, minimize variance