(GG) MoE vs. MLP on Tabular Data

2025-02-05

AI Summary

While MLPs and their ensembles achieve efficiency in tabular data modeling, they suffer from architectural stagnation; lightweight models struggle to balance performance and computational efficiency. Method: This paper proposes GG-MoE—a lightweight Mixture-of-Experts (MoE) model featuring Gumbel-Softmax gating for differentiable sparse routing, integrated with embedding layer optimization, parameter sharing, and sparse activation mechanisms to enhance generalization and efficiency. Results: Evaluated on 38 standard tabular datasets, GG-MoE consistently outperforms MLPs and classical MoE baselines, reducing parameter count by 40–65% and accelerating training and inference by 1.8–2.3×. It natively supports ensemble learning and scalable deployment. The core contribution lies in empirically validating that a structurally streamlined, controllably routed MoE paradigm delivers superior performance and practical utility for tabular learning.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In recent years, significant efforts have been directed toward adapting modern neural network architectures for tabular data. However, despite their larger number of parameters and longer training and inference times, these models often fail to consistently outperform vanilla multilayer perceptron (MLP) neural networks. Moreover, MLP-based ensembles have recently demonstrated superior performance and efficiency compared to advanced deep learning methods. Therefore, rather than focusing on building deeper and more complex deep learning models, we propose investigating whether MLP neural networks can be replaced with more efficient architectures without sacrificing performance. In this paper, we first introduce GG MoE, a mixture-of-experts (MoE) model with a Gumbel-Softmax gating function. We then demonstrate that GG MoE with an embedding layer achieves the highest performance across $38$ datasets compared to standard MoE and MLP models. Finally, we show that both MoE and GG MoE utilize significantly fewer parameters than MLPs, making them a promising alternative for scaling and ensemble methods.

Problem

• Efficient architectures for tabular data
• Performance comparison: MoE vs. MLP
• Reducing parameters in neural networks

Innovation

• Introduces GG MoE with Gumbel-Softmax gating
• GG MoE outperforms standard MoE and MLP
• MoE models use fewer parameters than MLPs