Revisit CP Tensor Decomposition: Statistical Optimality and Fast Convergence

2025-05-29

AI Summary

This paper addresses two fundamental limitations in CANDECOMP/PARAFAC (CP) tensor decomposition: the lack of statistical optimality guarantees under noise, non-orthogonality, and high rank, and the weak convergence theory for the alternating least squares (ALS) algorithm. To resolve these, we propose a joint framework integrating the Tensor Alternating Spectral Decomposition (TASD) initialization with ALS. We establish, for the first time, non-asymptotic minimax-optimal error bounds for CP decomposition under general order, dimension, and rank. We further characterize ALS’s two-stage convergence dynamics: quadratic convergence initially, followed by linear refinement; in the rank-one case, only 1–2 iterations suffice to achieve statistical optimality. Experiments demonstrate that TASD+ALS significantly improves estimation stability and accuracy in noisy settings. Our core contribution is a unified theoretical characterization of both the statistical limits and algorithmic convergence rates of CP decomposition, thereby filling a critical gap in high-dimensional tensor decomposition theory.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Canonical Polyadic (CP) tensor decomposition is a fundamental technique for analyzing high-dimensional tensor data. While the Alternating Least Squares (ALS) algorithm is widely used for computing CP decomposition due to its simplicity and empirical success, its theoretical foundation, particularly regarding statistical optimality and convergence behavior, remain underdeveloped, especially in noisy, non-orthogonal, and higher-rank settings. In this work, we revisit CP tensor decomposition from a statistical perspective and provide a comprehensive theoretical analysis of ALS under a signal-plus-noise model. We establish non-asymptotic, minimax-optimal error bounds for tensors of general order, dimensions, and rank, assuming suitable initialization. To enable such initialization, we propose Tucker-based Approximation with Simultaneous Diagonalization (TASD), a robust method that improves stability and accuracy in noisy regimes. Combined with ALS, TASD yields a statistically consistent estimator. We further analyze the convergence dynamics of ALS, identifying a two-phase pattern-initial quadratic convergence followed by linear refinement. We further show that in the rank-one setting, ALS with an appropriately chosen initialization attains optimal error within just one or two iterations.

Problem

• Analyzing statistical optimality of CP tensor decomposition
• Improving ALS convergence in noisy, non-orthogonal settings
• Developing robust initialization for accurate tensor decomposition

Innovation

• Tucker-based Approximation with Simultaneous Diagonalization (TASD)
• Non-asymptotic minimax-optimal error bounds
• Two-phase convergence dynamics analysis