MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper

2025-08-31

AI Summary

Traditional pretraining-fine-tuning paradigms are constrained by fixed parameter spaces, limiting their capacity to dynamically adapt to diverse and non-stationary data distributions. To address this, we propose Mixture of Expert Prompt Tuning (MEPT), which models deep networks as manifold mappers, integrating prompt tuning, Mixture-of-Experts (MoE) architecture, and dynamic routing. MEPT enables adaptive expert selection and neural activation path optimization at the prompt level. Grounded in manifold learning theory, it enhances model representation and generalization over complex data manifolds. On the SuperGLUE benchmark, MEPT achieves an average accuracy gain of 1.94% over state-of-the-art parameter-efficient methods, while reducing activated prompt tokens by 79.25%. Visualization analyses confirm expert-driven, heterogeneous neural pathway activation.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Considering deep neural networks as manifold mappers, the pretrain-then-fine-tune paradigm can be interpreted as a two-stage process: pretrain establishes a broad knowledge base, and fine-tune adjusts the model parameters to activate specific neural pathways to align with the target manifold. Although prior fine-tuning approaches demonstrate success, their rigid parameter space limits their ability to dynamically activate appropriate neural pathways, rendering them ill-equipped to adapt flexibly to the diverse and evolving data distributions. In light of this view, we propose a novel approach, Mixture of Expert Prompt Tuning (MEPT), as an effective and efficient manifold-mapping framework. MEPT leverages the Mixture of Experts architecture by integrating multiple prompt experts to adaptively learn diverse and non-stationary data distributions. Empirical evaluations demonstrate that MEPT outperforms several state-of-the-art parameter efficient baselines on SuperGLUE, achieving notable improvements in mean accuracy (e.g., 1.94%) while significantly reducing activated prompts by 79.25%. The effectiveness of MEPT is further supported by theoretical insights from manifold learning and validated through neural activation pathway visualization results. Our code is avaliable at https://github.com/runtsang/MEPT.

Problem

• Adapts to diverse data distributions dynamically
• Overcomes rigid parameter space limitations in fine-tuning
• Activates appropriate neural pathways for evolving data

Innovation

• Mixture of Expert Prompt Tuning framework
• Adaptively learns diverse data distributions
• Integrates multiple prompt experts architecture