Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding

2026-07-16

AI Summary

Existing multimodal large language models rely on a single visual prior, limiting their ability to effectively support diverse spatial understanding tasks. To address this, this work proposes the ViPS framework, which systematically reveals—for the first time—the complementary nature of multiple visual priors. ViPS introduces lightweight prior proxies and a dynamic fusion mechanism that enables context-aware collaborative injection of heterogeneous priors. By moving beyond the constraints of a single expert representation, the method achieves significant performance gains over current state-of-the-art approaches across multiple challenging benchmarks involving complex spatial reasoning and 3D understanding, establishing new best results.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated substantial promise in spatial understanding. Existing works typically incorporate prior knowledge extracted from a pre-trained foundation model to further enhance the spatial awareness of MLLMs. In this paper, we first reveal that when integrating diverse foundation models into MLLMs, different models provide complementary spatial priors that benefit different tasks. Motivated by this, we propose $\textbf{ViPS}$, a novel multi-model prior framework designed to fully unleash the potential of incorporating multiple $\textbf{Vi}$sual $\textbf{P}$riors from diverse models into MLLMs for $\textbf{S}$patial understanding. Specifically, ViPS introduces an Efficient Prior Proxy to generate multiple foundational priors with minimal inference overhead, and a Dynamic Prior Fusion mechanism to achieve harmonious and context-aware prior fusion and injection from the prior proxies. Extensive experiments demonstrate that ViPS successfully harmonizes diverse visual priors, establishing new state-of-the-art performance across multiple complex spatial reasoning and 3D spatial understanding benchmarks. Project page: https://visual-ai.github.io/vips

Problem

• Multimodal Large Language Models
• Visual Priors
• Spatial Understanding
• Foundation Models
• Prior Integration

Innovation

• multimodal large language models
• visual priors
• spatial understanding
• dynamic fusion
• efficient proxy