MuonAll: Muon Variant for Efficient Finetuning of Large Language Models

2025-11-08

AI Summary

Existing Muon optimizers only adapt a subset of model parameters, limiting their efficiency in fine-tuning large language models (LLMs). To address this, we propose MuonAll—the first Muon-based optimizer that uniformly incorporates *all* model parameters. It employs a systematic 2D matrix reparameterization technique to enable efficient gradient updates, significantly broadening applicability to open-source pre-trained language models with up to one billion parameters. The method supports distributed training and maintains compatibility with mainstream deep learning frameworks. Extensive experiments across model scales (100M–1B) and standard benchmarks (GLUE, Evaluating LLMs) demonstrate that MuonAll achieves performance on par with AdamW, validating its viability as a general-purpose drop-in replacement optimizer. The implementation is publicly available.

Technology Category

• Machine Learning: Optimization
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Muon optimizer has demonstrated robust results in pretraining of language models but its performance in finetuning of existing public pretrained models is not yet explored. Currently, Muon is used along with AdamW introducing a scope of improvement for adopting all parameters inside Muon. We introduce MuonAll, which incorporates all the parameters inside Muon by transforming into 2D matrices. We conduct extensive finetuning experiments across publicly available language models with model sizes upto half billion parameters. Muon and MuonAll perform at par with AdamW across major benchmarks, highlighting their effectiveness as alternative optimizers. We open-source the distributed implementations of Muon and MuonAll, available at https://github.com/Saurabh750/optimizer

Problem

• Exploring Muon optimizer's effectiveness in finetuning pretrained language models
• Improving parameter integration by transforming them into 2D matrices
• Evaluating performance across models up to half billion parameters

Innovation

• MuonAll transforms all parameters into 2D matrices
• It incorporates all parameters inside the Muon optimizer
• Performs comparably to AdamW across major benchmarks