Divide, Weight, and Route: Difficulty-Aware Optimization with Dynamic Expert Fusion for Long-tailed Recognition

2025-08-27

AI Summary

Long-tailed visual recognition suffers from both class imbalance and intrinsic differences in classification difficulty; conventional reweighting methods neglect hard-to-learn classes. To address this, we propose a difficulty-aware dynamic expert collaboration framework. First, we quantify per-class difficulty via uncertainty estimation and historical performance modeling, enabling adaptive loss weighting. Second, we introduce a decentralized expert routing mechanism: each expert is equipped with a dedicated out-of-distribution (OOD) detector and autonomously routes inference based on local confidence scores—eliminating the need for a centralized router. Our approach integrates mixture-of-experts architecture, uncertainty modeling, historical analysis, and end-to-end joint training. Experiments on standard long-tailed benchmarks demonstrate significant improvements in overall accuracy, particularly for rare and inherently difficult classes. These results validate the effectiveness and generalizability of difficulty-aware weighting and decentralized routing.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Long-tailed visual recognition is challenging not only due to class imbalance but also because of varying classification difficulty across categories. Simply reweighting classes by frequency often overlooks those that are intrinsically hard to learn. To address this, we propose extbf{DQRoute}, a modular framework that combines difficulty-aware optimization with dynamic expert collaboration. DQRoute first estimates class-wise difficulty based on prediction uncertainty and historical performance, and uses this signal to guide training with adaptive loss weighting. On the architectural side, DQRoute employs a mixture-of-experts design, where each expert specializes in a different region of the class distribution. At inference time, expert predictions are weighted by confidence scores derived from expert-specific OOD detectors, enabling input-adaptive routing without the need for a centralized router. All components are trained jointly in an end-to-end manner. Experiments on standard long-tailed benchmarks demonstrate that DQRoute significantly improves performance, particularly on rare and difficult classes, highlighting the benefit of integrating difficulty modeling with decentralized expert routing.

Problem

• Addresses long-tailed recognition with class imbalance and varying difficulty
• Proposes difficulty-aware optimization using prediction uncertainty and historical performance
• Introduces dynamic expert fusion with input-adaptive routing for specialized processing

Innovation

• Difficulty-aware adaptive loss weighting for training
• Mixture-of-experts design specializing in class distribution regions
• Confidence-based expert routing using OOD detectors without central router