Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism

2026-03-21

AI Summary

This work addresses the challenge of unreliable visual recognition in scenarios characterized by high inter-class similarity, significant scale variation, and limited computational resources. To this end, we propose the DS-MoE framework, which integrates a distilled large language model with a sparse Mixture-of-Experts (MoE) architecture. Our approach employs a text-guided dynamic routing mechanism to enable semantics-driven expert activation and incorporates a lightweight MobileSAM encoder for multi-scale defect awareness, thereby achieving precise alignment between text–visual semantics and defect patterns. Evaluated on the BBMP, aluminum foil, and PCB datasets, DS-MoE outperforms YOLOv8 and YOLOX by 13.9, 1.4, and 2.0 percentage points in mAP@0.5:0.95, respectively, while also delivering notable improvements in both precision and recall.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

High inter-class similarity, extreme scale variation, and limited computational budgets hinder reliable visual recognition across diverse real-world data. Existing vision-centric and cross-modal approaches often rely on rigid fusion mechanisms and heavy annotation pipelines, leading to sub-optimal generalization. We propose the Distilled Large Language Model (LLM)-Driven Sparse Mixture-of-Experts (DS-MoE) framework, which integrates text-guided dynamic routing and lightweight multi-scale comprehension. The DS-MoE framework dynamically aligns textual semantics with defect-specific visual patterns through a sparse MoE architecture, where task-relevant experts are adaptively activated based on semantic relevance, resolving inter-class ambiguity. A lightweight MobileSAM encoder enables real-time inference while preserving multi-scale defect details. Extensive experiments on PCB, aluminum foil, and mold defect datasets demonstrate that our framework achieves superior performance compared to existing pure vision models. \textbf{DS-MoE} surpasses YOLOv8/YOLOX with gains of +13.9, +1.4, and +2.0 pp mAP@ 0.5:0.95 on BBMP, aluminum, and PCB, respectively, while also improving precision and recall.

Problem

• inter-class similarity
• scale variation
• computational budgets
• visual recognition
• real-world data

Innovation

• Distilled LLM
• Sparse Mixture-of-Experts
• Text-guided dynamic routing
• Lightweight multi-scale encoding
• Inter-class ambiguity resolution