Learnability and Competition in High-Dimensional Multi-Component ICA

2026-05-08

AI Summary

This work addresses the lack of theoretical understanding regarding the coupling effects induced by simultaneous learning and orthogonalization in high-dimensional multicomponent independent component analysis (ICA) and their impact on learnability. By establishing a mean-field theory in the high-dimensional limit, the study characterizes the dynamic evolution of learning directions relative to true components through an ordinary differential equation system governing overlap matrices. It reveals, for the first time, two distinct phases—decoupling and competition—driven by initialization. The theory delineates how the interplay among learning rate, higher-order data moments, and initialization determines the learnability boundary and competition conditions, predicting prolonged convergence times and a stepwise phenomenon in the number of recoverable components. Validated through high-dimensional asymptotics and stochastic process limits, the proposed framework accurately reproduces trajectory dynamics, phase transitions, and the discrete influence of learning rate on recovery performance in both synthetic and hyperspectral remote sensing data.

Abstract

Independent Component Analysis (ICA) is a foundational tool for unsupervised representation learning, yet its high-dimensional theory remains largely limited to single-component recovery. We develop an asymptotically exact mean-field theory for multi-component online ICA, capturing the coupling induced by simultaneous learning and orthogonalization. In the high-dimensional limit, the joint empirical distribution of learned estimates and ground-truth components converges to a deterministic process, yielding a closed ODE system for the overlap matrix between learned directions and true components. This characterization reveals a genuinely multi-component, initialization-driven phase structure: a decoupled regime, where estimates align with distinct components and evolve nearly independently, and a competition regime, where overlapping initializations induce orthogonality-driven conflicts, slow reorientation, and delayed convergence. Our steady-state analysis gives explicit learnability boundaries and competition conditions linking step size, data moments, and initialization. These conditions show that larger higher-order moments and competition shrink the stable learning-rate window, increase convergence times, and predict a staircase phenomenon in which the number of recoverable components changes discretely with the learning rate. Experiments on synthetic data and hyperspectral remote sensing data validate the predicted trajectories and phase behavior.

Problem

• Independent Component Analysis
• high-dimensional
• multi-component
• learnability
• competition

Innovation

• multi-component ICA
• mean-field theory
• high-dimensional asymptotics
• learning dynamics
• phase transition