MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein

2026-06-28

AI Summary

This work addresses a structural blind spot in multimodal large language models that impairs their ability to transfer linguistic semantic relations to visual scenes, leading to failures in relational reasoning. To overcome this limitation, the authors propose the MIRROR framework, which introduces—for the first time—the semi-inverse Gromov–Wasserstein problem and derives its closed-form solution to enable parameter-free, efficient alignment of linguistic relational structures with visual representations. By integrating optimal transport–driven geometric regularization with a Transformer-oriented strategy for stable extraction at the layer, head, and token levels, the method significantly enhances relational consistency between vision and language while preserving strong performance on general-purpose tasks.

Abstract

Multimodal Large Language Models (MLLMs) inherit rich relational priors from their language backbones, yet often fail when asked to apply these relationships in visual contexts. We trace this failure to a structural blind spot: projection-based alignment trains each visual token to carry the right semantics, but never asks whether the relationships between concepts survive the crossing from language to vision. To address this, we propose MIRROR (Mapping Inter-concept Relations from language to visual Representation via Optimal-transport-based Regularization), a geometric regularization framework that transfers relational priors from language to vision by exploiting the rich relational structure encoded in language representations. Specifically, we derive a surrogate loss from the proposed Semi-Inverse Gromov-Wasserstein (SI-GW) problem, an inverse geometric problem that aligns visual representations with language-derived relational priors. We show that this formulation admits a unique closed-form solution that prescribes the ideal visual relational structure implied by language geometry and cross-modal coupling. The structure of the formulation also enables efficient computation, making it applicable to long token sequences. Applying SI-GW inside decoder-only Transformers requires careful design. We introduce targeted strategies at the layer, head, and token levels to ensure stable extraction without additional parameters or inference cost. MIRROR improves relational consistency while preserving performance on general vision-language tasks.

Problem

• Multimodal Large Language Models
• Semantic Relations
• Vision-Language Alignment
• Relational Priors
• Cross-modal Transfer

Innovation

• Gromov-Wasserstein
• relational alignment
• multimodal learning
• geometric regularization
• vision-language models