Structural Properties, Cycloid Trajectories and Non-Asymptotic Guarantees of EM Algorithm for Mixed Linear Regression

2025-11-07

AI Summary

This work addresses the weak theoretical foundation of the Expectation-Maximization (EM) algorithm for two-component mixed linear regression (2MLR), where both mixing weights and regression parameters are unknown, and the non-asymptotic convergence behavior and iteration trajectory remain poorly understood. We propose a trajectory-based analytical framework. First, we reveal that the EM update path exhibits a cycloidal structure: it is exactly cycloidal under zero noise and asymptotically approaches a cycloid at high signal-to-noise ratios (SNR). Second, we characterize a phase transition between linear and quadratic convergence governed by the initial angle. Third, by deriving an explicit closed-form update rule valid across the full SNR range—combined with suboptimal angular recurrence modeling and finite-sample error decomposition—we establish the first non-asymptotic convergence guarantee for arbitrary initializations. This work provides a geometric interpretation of EM and a verifiable, non-asymptotic convergence theory for 2MLR.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations

Abstract

This work investigates the structural properties, cycloid trajectories, and non-asymptotic convergence guarantees of the Expectation-Maximization (EM) algorithm for two-component Mixed Linear Regression (2MLR) with unknown mixing weights and regression parameters. Recent studies have established global convergence for 2MLR with known balanced weights and super-linear convergence in noiseless and high signal-to-noise ratio (SNR) regimes. However, the theoretical behavior of EM in the fully unknown setting remains unclear, with its trajectory and convergence order not yet fully characterized. We derive explicit EM update expressions for 2MLR with unknown mixing weights and regression parameters across all SNR regimes and analyze their structural properties and cycloid trajectories. In the noiseless case, we prove that the trajectory of the regression parameters in EM iterations traces a cycloid by establishing a recurrence relation for the sub-optimality angle, while in high SNR regimes we quantify its discrepancy from the cycloid trajectory. The trajectory-based analysis reveals the order of convergence: linear when the EM estimate is nearly orthogonal to the ground truth, and quadratic when the angle between the estimate and ground truth is small at the population level. Our analysis establishes non-asymptotic guarantees by sharpening bounds on statistical errors between finite-sample and population EM updates, relating EM's statistical accuracy to the sub-optimality angle, and proving convergence with arbitrary initialization at the finite-sample level. This work provides a novel trajectory-based framework for analyzing EM in Mixed Linear Regression.

Problem

• Analyzes EM algorithm convergence for mixed linear regression
• Characterizes cycloid trajectories in noiseless and noisy regimes
• Establishes non-asymptotic guarantees with arbitrary initialization

Innovation

• Derived explicit EM update expressions for unknown parameters
• Analyzed cycloid trajectories and structural EM properties
• Established non-asymptotic convergence guarantees with initialization