Expected Shortfall Regression via Optimization

2026-02-21

AI Summary

This study addresses the challenge that existing methods struggle to accurately characterize the influence of covariates on the tail distribution of a response variable, and that superquantile regression and Expected Shortfall (ES) regression often yield inconsistent estimates. To resolve this, the authors propose an optimization-based linear ES regression approach that avoids imposing additional assumptions on conditional quantiles and instead employs an implicit loss function to precisely model tail risk. The key innovation lies in explicitly distinguishing between superquantile and ES regression for the first time and introducing heterogeneity-adaptive weights to enhance estimation efficiency. By combining binning-based initial values with a tailored optimization algorithm, the method ensures consistency and asymptotic normality, while simulation studies demonstrate its superior performance over existing approaches across various settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

To provide a comprehensive summary of the tail distribution, the expected shortfall is defined as the average over the tail above (or below) a certain quantile of the distribution. The expected shortfall regression captures the heterogeneous covariate-response relationship and describes the covariate effects on the tail of the response distribution. Based on a critical observation that the superquantile regression from the operations research literature does not coincide with the expected shortfall regression, we propose and validate a novel optimization-based approach for the linear expected shortfall regression, without additional assumptions on the conditional quantile models. While the proposed loss function is implicitly defined, we provide a prototype implementation of the proposed approach with some initial expected shortfall estimators based on binning techniques. With practically feasible initial estimators, we establish the consistency and the asymptotic normality of the proposed estimator. The proposed approach achieves heterogeneity-adaptive weights and therefore often offers efficiency gain over existing linear expected shortfall regression approaches in the literature, as demonstrated through simulation studies.

Problem

• Expected Shortfall Regression
• Tail Risk
• Superquantile Regression
• Heterogeneous Covariate Effects
• Distributional Tail

Innovation

• Expected Shortfall Regression
• Optimization-based Estimation
• Superquantile Regression
• Heterogeneity-adaptive Weights
• Asymptotic Normality