ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modeling

2026-06-30

AI Summary

Existing multi-objective masked image modeling approaches employ global loss weights, overlooking the spatial heterogeneity among image patches. This work proposes ExPLoRe, which, for the first time, leverages the routing weights of Soft Mixture-of-Experts (Soft MoE) as learnable per-patch loss coefficients and introduces a loss coupling mechanism that enables the routing to adaptively optimize each objective based on image content. The method facilitates content-aware multi-objective training within a Vision Transformer (ViT) backbone. Experimental results demonstrate that, on ImageNet-1K, the approach achieves linear probing and fine-tuning accuracies of 80.6% and 85.3%, respectively, with a ViT-Base model. Furthermore, it substantially outperforms non-MoE baselines on the ADE20K semantic segmentation task, reducing the mIoU gap by 2.5–2.9 points.

Abstract

Multi-objective masked image modeling (MIM) combines complementary learning signals (token distillation, CLS alignment, and pixel reconstruction) but existing methods weight these objectives with global scalars, ignoring spatial heterogeneity across patches. We present ExPLoRe (Expert Patch-Level Loss Routing), which repurposes Soft Mixture of Experts (MoE) dispatch weights as learned, per-patch loss coefficients. The key mechanism is loss-coupling: allowing loss gradients to flow through dispatch weights to the router enables content-dependent specialization, where different patches receive different emphases across objectives. A detach ablation confirms loss-coupling as the core mechanism, degrading performance by 1.6% when gradients are blocked. On ImageNet-1K with ViT-Base, ExPLoRe improves over non-MoE baselines on two objective combinations (Token+CLS: +0.5% k-NN, +4.4% linear probe; Token+Pixel: +2.2% k-NN), achieving 80.6% linear probe and 85.3% finetuning accuracy, competitive with published methods. For downstream transfer, we develop adaptation recipes (Freeze Routing, Expert Dropout, and Freeze Attention) that improve MoE finetuning by +1.5% over the vanilla MoE, and close a 2.5--2.9 mIoU segmentation gap so that MoE models match or exceed non-MoE baselines on ADE20K.

Problem

• masked image modeling
• multi-objective learning
• spatial heterogeneity
• patch-level weighting
• loss routing

Innovation

• Patch-Level Loss Routing
• Loss-Coupling
• Soft Mixture of Experts
• Multi-Objective Masked Image Modeling
• Content-Dependent Specialization