When to use what Schatten-$p$ norm in deep learning?

2026-06-13

AI Summary

This study addresses the empirical contradictions surrounding the effectiveness of Schatten-$p$ norm optimizers such as Muon in deep learning by rigorously characterizing their performance conditions. Integrating the SODA optimization framework, geometric analysis of Schatten-$p$ spaces, and noise robustness theory, the work systematically investigates the optimization geometry under varying training regimes—including dimensionality and Chinchilla scaling laws. It theoretically establishes, for the first time, that smaller values of $p$ are preferable in low-dimensional settings and under Chinchilla scaling, and derives a general batch-size scaling rule valid for any $p$. Furthermore, it delineates the performance boundary where Schatten-$\infty$ methods excel. This unified analysis explains empirical observations such as Muon’s lack of need for warmup and its preference for large batch sizes, offering clear guidelines for norm selection in practice.

Abstract

Schatten-$\infty$ based optimizers such as Muon have shown promising empirical performance, but there remains seemingly conflicting observations regarding whether they are beneficial. We resolve this conflict by showing that the conclusion is regime dependent. Even when the objective is smooth in the Schatten-$\infty$ geometry, smaller Schatten-$p$ geometries can be optimal, specifically in the low-dimensional regime, which we show includes Chinchilla scaling. This conclusion follows from a new noise-robust acceleration result for the SODA framework for $p>2$. The same analysis explains why Muon-like methods do not require warmup, why they naturally favor large batches, and yields a batch size scaling rule for arbitrary $p$.

Problem

• Schatten-p norm
• deep learning
• optimization
• regime dependence
• batch size scaling

Innovation

• Schatten-p norm
• SODA framework
• noise-robust acceleration
• regime-dependent optimization
• batch size scaling