Schattor: Schatten-family methods for deep learning optimization

2026-06-14

AI Summary

This work addresses key challenges in deep learning optimization—such as heterogeneous parameter structures, high gradient noise, and highly non-convex loss landscapes—by introducing the Schattor family of methods, a novel adaptive first-order optimization framework grounded in Schatten norms. Schattor is the first to incorporate Schatten norms into deep learning optimization, unifying stochastic gradient descent (SGD) and matrix-adaptive optimizers like Muon within a single theoretical framework, while enabling multi-block adaptive balancing. Leveraging matrix martingale concentration inequalities and stochastic matrix optimization analysis, the authors establish dimension-free stationarity guarantees and prove dimension-independent convergence for stochastic matrix optimization problems. This significantly generalizes and enhances the performance and applicability of existing optimizers.

Abstract

Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm design and theoretical analysis. Motivated by the limitations of SGD and the success of adaptive optimizers, we propose {\it Schattor}, a family of adaptive first-order methods based on Schatten norms. Schattor unifies SGD and the recently proposed matrix-variate adaptive optimizer Muon within a single Schatten-norm-based framework. We establish dimension-free stationarity guarantees for methods in the Schattor family for stochastic matrix optimization problems via a novel matrix martingale moment bound. We also develop multi-block extensions that adaptively balance block-wise optimization progress and prove dimension-free stationarity guarantees in this more general setting.

Problem

• deep learning optimization
• heterogeneous parameter structures
• noisy gradients
• nonconvex landscapes
• adaptive optimizers

Innovation

• Schatten norms
• adaptive optimization
• matrix martingale
• dimension-free guarantees
• multi-block optimization