Estimation of Toeplitz Covariance Matrices using Overparameterized Gradient Descent

2025-11-03

AI Summary

To address the challenges of local optima and poor convergence guarantees in conventional optimization for Toeplitz-structured covariance matrix estimation, this paper proposes an overparameterized gradient descent framework. Specifically, the covariance matrix is modeled as a sum of complex sinusoids, and optimization is accelerated via a dual learning rate strategy that decouples amplitude and frequency updates. Theoretically, under mild overparameterization and fixed-frequency assumptions, the Gaussian log-likelihood objective exhibits an asymptotically benign optimization landscape, ensuring global convergence from random initialization. Empirically, the method achieves state-of-the-art estimation accuracy in challenging regimes—including low signal-to-noise ratios and limited samples—while maintaining algorithmic simplicity and strong scalability. The key contributions are: (i) the first global convergence guarantee for overparameterized gradient descent in structured covariance estimation, and (ii) a practical, frequency-aware optimization mechanism that significantly improves convergence speed and robustness.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

We consider covariance estimation under Toeplitz structure. Numerous sophisticated optimization methods have been developed to maximize the Gaussian log-likelihood under Toeplitz constraints. In contrast, recent advances in deep learning demonstrate the surprising power of simple gradient descent (GD) applied to overparameterized models. Motivated by this trend, we revisit Toeplitz covariance estimation through the lens of overparameterized GD. We model the $P imes P$ covariance as a sum of $K$ complex sinusoids with learnable parameters and optimize them via GD. We show that when $K = P$, GD may converge to suboptimal solutions. However, mild overparameterization ($K = 2P$ or $4P$) consistently enables global convergence from random initializations. We further propose an accelerated GD variant with separate learning rates for amplitudes and frequencies. When frequencies are fixed and only amplitudes are optimized, we prove that the optimization landscape is asymptotically benign and any stationary point recovers the true covariance. Finally, numerical experiments demonstrate that overparameterized GD can match or exceed the accuracy of state-of-the-art methods in challenging settings, while remaining simple and scalable.

Problem

• Estimating Toeplitz covariance matrices using overparameterized gradient descent methods
• Achieving global convergence through mild overparameterization in covariance estimation
• Optimizing complex sinusoid parameters to recover true covariance structure

Innovation

• Overparameterized gradient descent for Toeplitz covariance estimation
• Accelerated gradient descent with separate learning rates
• Modeling covariance as sum of complex sinusoids