High-Dimensional Precision Matrix Quadratic Forms: Estimation Framework for $p>n$

2026-01-07

AI Summary

This work addresses the failure of conventional quadratic forms involving precision matrices in high-dimensional settings where the feature dimension \(p\) exceeds the sample size \(n\), a regime plagued by rank deficiency and excessive complexity. The authors propose a novel framework based on spectral moment representations and constrained optimization, which, under mild moment conditions, achieves the first consistent estimator for such quadratic forms. This approach overcomes fundamental limitations of existing methods and provides a unified treatment for a broad class of high-dimensional statistics. Both theoretical analysis and numerical simulations demonstrate that the proposed method substantially outperforms traditional approaches in applications including optimal portfolio Sharpe ratio estimation and multiple correlation coefficients in regression, effectively resolving the estimation breakdown inherent to the \(p > n\) setting.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Non-convex Optimization
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

We propose a novel estimation framework for quadratic functionals of precision matrices in high-dimensional settings, particularly in regimes where the feature dimension $p$ exceeds the sample size $n$. Traditional moment-based estimators with bias correction remain consistent when $pn$, highlighting a fundamental distinction between the two regimes due to rank deficiency and high-dimensional complexity. Our approach resolves these issues by combining a spectral-moment representation with constrained optimization, resulting in consistent estimation under mild moment conditions. The proposed framework provides a unified approach for inference on a broad class of high-dimensional statistical measures. We illustrate its utility through two representative examples: the optimal Sharpe ratio in portfolio optimization and the multiple correlation coefficient in regression analysis. Simulation studies demonstrate that the proposed estimator effectively overcomes the fundamental $p>n$ barrier where conventional methods fail.

Problem

• precision matrix
• quadratic forms
• high-dimensional statistics
• p > n
• covariance estimation

Innovation

• precision matrix
• quadratic forms
• high-dimensional statistics
• p > n
• constrained optimization