Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs

2025-08-27

AI Summary

Large language models frequently suffer from context disconnection—generating outputs inconsistent with input contexts—in context-dependent tasks. This paper presents the first empirical discovery of “context fidelity specialization” among experts in Mixture-of-Experts (MoE) models: certain experts inherently excel at modeling contextual dependencies. To exploit this phenomenon, we propose Router Lens, an attention-based analysis method that identifies context-faithful experts. Building upon this insight, we design CEFT—a lightweight selective fine-tuning framework that updates only the parameters of context-faithful experts, drastically reducing computational overhead. Extensive experiments across multiple benchmarks and diverse MoE architectures demonstrate that CEFT matches or surpasses full-parameter fine-tuning in performance, confirming its effectiveness and efficiency. Our core contribution lies in uncovering functional specialization among MoE experts and establishing the first context-fidelity-oriented optimization paradigm for LLMs.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Context faithfulness is essential for reliable reasoning in context-dependent scenarios. However, large language models often struggle to ground their outputs in the provided context, resulting in irrelevant responses. Inspired by the emergent expert specialization observed in mixture-of-experts architectures, this work investigates whether certain experts exhibit specialization in context utilization, offering a potential pathway toward targeted optimization for improved context faithfulness. To explore this, we propose Router Lens, a method that accurately identifies context-faithful experts. Our analysis reveals that these experts progressively amplify attention to relevant contextual information, thereby enhancing context grounding. Building on this insight, we introduce Context-faithful Expert Fine-Tuning (CEFT), a lightweight optimization approach that selectively fine-tunes context-faithful experts. Experiments across a wide range of benchmarks and models demonstrate that CEFT matches or surpasses the performance of full fine-tuning while being significantly more efficient.

Problem

• Identify experts specializing in context faithfulness
• Enhance grounding of outputs in provided context
• Optimize context utilization via selective fine-tuning

Innovation

• Router Lens identifies context-faithful experts
• CEFT selectively fine-tunes specialized experts
• Lightweight optimization enhances context attention