End-to-End Large Portfolio Optimization for Variance Minimization with Neural Networks through Covariance Cleaning

2025-07-02

AI Summary

This paper addresses ill-conditioning of the covariance matrix, bias in marginal volatility estimation, and poor generalizability in large-scale portfolio variance minimization. We propose a rotation-invariant, end-to-end neural network framework that jointly models lagged return structures and nonlinear shrinkage regularization of covariance eigenvalues and volatilities, while explicitly embedding the analytical solution structure of the global minimum-variance (GMV) portfolio—yielding an interpretable, modular architecture. Key contributions include: (i) zero-shot generalization across varying asset counts—adapting seamlessly to different stock universes without retraining; and (ii) integrated covariance denoising with rotation-invariant design, substantially enhancing out-of-sample stability. Empirical evaluation over 2000–2024 demonstrates consistent superiority over benchmarks in reducing volatility and maximum drawdown, improving Sharpe ratios, maintaining robustness under transaction costs and market stress, and supporting extensions such as long-only constraints.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Learning & Optimization for CV
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

We develop a rotation-invariant neural network that provides the global minimum-variance portfolio by jointly learning how to lag-transform historical returns and how to regularise both the eigenvalues and the marginal volatilities of large equity covariance matrices. This explicit mathematical mapping offers clear interpretability of each module's role, so the model cannot be regarded as a pure black-box. The architecture mirrors the analytical form of the global minimum-variance solution yet remains agnostic to dimension, so a single model can be calibrated on panels of a few hundred stocks and applied, without retraining, to one thousand US equities-a cross-sectional jump that demonstrates robust out-of-sample generalisation. The loss function is the future realized minimum portfolio variance and is optimized end-to-end on real daily returns. In out-of-sample tests from January 2000 to December 2024 the estimator delivers systematically lower realised volatility, smaller maximum drawdowns, and higher Sharpe ratios than the best analytical competitors, including state-of-the-art non-linear shrinkage. Furthermore, although the model is trained end-to-end to produce an unconstrained (long-short) minimum-variance portfolio, we show that its learned covariance representation can be used in general optimizers under long-only constraints with virtually no loss in its performance advantage over competing estimators. These gains persist when the strategy is executed under a highly realistic implementation framework that models market orders at the auctions, empirical slippage, exchange fees, and financing charges for leverage, and they remain stable during episodes of acute market stress.

Problem

• Develop neural network for global minimum-variance portfolio optimization
• Clean and regularize large equity covariance matrices effectively
• Achieve robust out-of-sample performance with interpretable modules

Innovation

• Rotation-invariant neural network for portfolio optimization
• Joint learning of lag-transform and covariance regularization
• End-to-end training with future realized variance loss