Divide, Weight, and Route: Difficulty-Aware Optimization with Dynamic Expert Fusion for Long-tailed Recognition
Long-tailed visual recognition suffers from both class imbalance and intrinsic differences in classification difficulty; conventional reweighting methods neglect hard-to-learn classes. To address this, we propose a difficulty-aware dynamic expert collaboration framework. First, we quantify per-class difficulty via uncertainty estimation and historical performance modeling, enabling adaptive loss weighting. Second, we introduce a decentralized expert routing mechanism: each expert is equipped with a dedicated out-of-distribution (OOD) detector and autonomously routes inference based on local confidence scores—eliminating the need for a centralized router. Our approach integrates mixture-of-experts architecture, uncertainty modeling, historical analysis, and end-to-end joint training. Experiments on standard long-tailed benchmarks demonstrate significant improvements in overall accuracy, particularly for rare and inherently difficult classes. These results validate the effectiveness and generalizability of difficulty-aware weighting and decentralized routing.