Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning

2026-05-09

AI Summary

This work addresses catastrophic forgetting in continual learning for large language models by proposing a spectral-norm-based orthogonal gradient projection mechanism. It introduces, for the first time, a non-Euclidean continual learning framework tailored for matrix parameters by integrating the spectral norm geometry of the Muon optimizer with orthogonality constraints. The method effectively balances model stability and plasticity while maintaining computational scalability through spectral-norm-constrained optimization, dual iterative solving, Newton–Schulz matrix sign approximation, and orthogonal momentum updates. Experimental results demonstrate that the proposed approach significantly outperforms existing baselines across diverse architectures and benchmark tasks, substantially mitigating catastrophic forgetting.

Abstract

A central challenge in continual learning for large language models (LLMs) is catastrophic forgetting, where adapting to new tasks can substantially degrade performance on previously learned ones. Existing projection-based methods mitigate such interference by restricting parameter updates to subspaces that are orthogonal to directions associated with past tasks. However, these methods are typically formulated under Euclidean parameter geometry, with update magnitudes and projections governed by the Frobenius norm. The recent empirical success of the Muon optimizer, which applies orthogonalized matrix updates and admits a spectral-norm interpretation, suggests that Frobenius geometry may not be the most effective choice for matrix-valued LLM parameters. Motivated by this observation, we propose Muon-OGD, a spectral-norm-aware continual learning framework that integrates Muon-style operator-norm geometry with orthogonal projection constraints. Our method formulates each update as a spectral-norm-constrained optimization problem with linear non-interference constraints, and solves it efficiently through dual iterations and Newton--Schulz matrix-sign approximations. By applying orthogonalized momentum updates that avoid protected directions associated with prior tasks, Muon-OGD aims to improve the stability--plasticity trade-off in sequential LLM adaptation. We evaluate the proposed method on standard continual learning benchmarks, TRACE, and domain-specific Coding--Math--Medical curricula using both encoder--decoder and decoder-only architectures. Empirically, Muon-OGD consistently improves over sequential fine-tuning and competitive orthogonal-gradient baselines, while remaining computationally scalable. These results suggest that spectral-norm-aware update geometry provides a practical and effective alternative to Frobenius-norm projection for continual learning in LLMs.

Problem

• continual learning
• catastrophic forgetting
• large language models
• orthogonal projection
• spectral norm

Innovation

• spectral norm
• orthogonal projection
• continual learning
• large language models
• Muon optimizer