Benchmarking Optimizers for MLPs in Tabular Deep Learning

2026-04-16

AI Summary

This study addresses the lack of systematic evaluation of optimizer selection in deep learning for tabular data, despite the widespread adoption of MLPs and AdamW. Under a unified experimental protocol, the authors present the first comprehensive benchmark comparing multiple optimizers for training MLPs across diverse tabular datasets. Their results demonstrate that the Muon optimizer consistently outperforms AdamW in most scenarios. Furthermore, integrating exponential moving average (EMA) of weights significantly enhances AdamW’s performance on standard MLP architectures. This work provides empirical evidence and practical guidance for optimizer selection in tabular data learning, challenging conventional defaults and highlighting effective alternatives.

Technology Category

• Machine Learning: Optimization
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

MLP is a heavily used backbone in modern deep learning (DL) architectures for supervised learning on tabular data, and AdamW is the go-to optimizer used to train tabular DL models. Unlike architecture design, however, the choice of optimizer for tabular DL has not been examined systematically, despite new optimizers showing promise in other domains. To fill this gap, we benchmark \Noptimizers optimizers on \Ndatasets tabular datasets for training MLP-based models in the standard supervised learning setting under a shared experiment protocol. Our main finding is that the Muon optimizer consistently outperforms AdamW, and thus should be considered a strong and practical choice for practitioners and researchers, if the associated training efficiency overhead is affordable. Additionally, we find exponential moving average of model weights to be a simple yet effective technique that improves AdamW on vanilla MLPs, though its effect is less consistent across model variants.

Problem

• tabular deep learning
• MLP
• optimizer benchmarking
• AdamW
• training optimization

Innovation

• optimizer benchmarking
• tabular deep learning
• Muon optimizer
• exponential moving average
• MLP