Task-Instructed Causal Routing of Vision Foundation Models for Multi-Task Learning

2026-06-14

AI Summary

Existing monolithic vision foundation models struggle to meet the diverse representational demands of multitask dense prediction due to inductive biases introduced during pretraining, leading to fragmented knowledge. To address this, this work proposes TIGER, a framework that, without updating any model parameters, leverages natural language task instructions to guide token-level routing to heterogeneous experts and aligns their causal contributions through a counterfactual causal loss for adaptive feature fusion. Evaluated on NYUD-v2 and Pascal Context benchmarks, TIGER significantly outperforms current state-of-the-art methods, demonstrating its effectiveness in enhancing multitask dense prediction performance.

Abstract

Vision foundation models (VFMs) have demonstrated strong robustness and transferability across a wide range of visual tasks. However, each model typically encodes strong inductive biases shaped by its pre-training objective and data domain, resulting in fragmented yet complementary visual knowledge. As a result, a single model often struggles to capture the diverse visual representations required across multiple dense prediction tasks. To address this limitation, we propose TIGER (Task-Instruction-Guided Expert Routing), a framework that coordinates multiple heterogeneous VFMs for multi-task dense prediction. Instead of naively aggregating expert features, TIGER leverages natural-language task instructions to guide a routing network that assigns token-level expert weights conditioned on task semantics, enabling adaptive integration of complementary expert features. TIGER further introduces a counterfactual loss that aligns routing decisions with each expert's causal contribution by measuring prediction changes when experts are excluded, encouraging more reliable and interpretable routing. We evaluate TIGER on two multi-task dense prediction benchmarks, NYUD-v2 and Pascal Context, where it consistently outperforms recent multi-task learning baselines while keeping all VFMs frozen. These results demonstrate that combining instruction-guided expert routing with counterfactual causal alignment enables effective coordination of heterogeneous vision foundation models.

Problem

• Vision Foundation Models
• Multi-Task Learning
• Dense Prediction
• Inductive Bias
• Expert Routing

Innovation

• expert routing
• vision foundation models
• multi-task learning
• causal alignment
• instruction-guided