Metric-Guided Feature Fusion of Visual Foundation Models for Segmentation Tasks

2026-05-16

AI Summary

This work addresses the limited performance of existing vision foundation models in instance-aware dense prediction tasks and the lack of effective mechanisms for fusing their complementary features. The authors propose a metric-guided feature fusion approach that, for the first time, leverages unsupervised criteria—structural consistency and edge fidelity—to evaluate and select superior encoder features directly in the feature space. By employing a primary-auxiliary fusion strategy, the method achieves efficient integration without requiring complex architectural modifications. Notably, it attains significant performance gains through only single-stage training, simultaneously enhancing both semantic representation and boundary localization accuracy across multiple dense prediction benchmarks.

Abstract

Although large-scale visual foundation models (VFMs) achieve remarkable performance in semantic understanding, they still underperform in instance-aware dense prediction tasks. They exhibit different biases in representation: for instance, promptable segmentation models (e.g., SAM2) focus on fine-grained region boundaries, while self-supervised models (e.g., DINOv3) emphasize object-level structure. This observation highlights the potential of combining complementary features from different VFMs to enhance downstream dense prediction tasks. However, naive multi-VFM fusion seldom leads to reliable gains, and interpretable principles for leveraging their complementary features are still underexplored. In this work, we propose a metric-guided approach that effectively selects and aggregates complementary features from different VFMs based on explicit assessment scores. Specifically, we design a suite of label-free metrics in feature space across two aspects, Structural Coherence and Edge Fidelity, to assess features of VFM encoders. Guided by these scores, we identify complementary edge-strong and structure-strong encoder pairs, and integrate them via a master-auxiliary fusion scheme. This feature fusion requires no complex architectural changes and is trained only in a single stage. Our model shows consistent performance gains across multiple dense prediction tasks compared with the baselines, with better object-level semantics and more accurately localized boundaries. The code is available at {https://github.com/gyc-code/metric-guided-fusion}.

Problem

• visual foundation models
• feature fusion
• dense prediction
• instance-aware segmentation
• complementary features

Innovation

• metric-guided fusion
• visual foundation models
• feature aggregation
• dense prediction
• label-free evaluation