Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis

2026-07-11

AI Summary

This work addresses the limited generalization of large multimodal models (LMMs) to unseen visual modalities beyond standard RGB inputs. To overcome this, the authors propose VVM-Tuning, a training framework that synthesizes diverse visual modalities to encourage the model to disentangle scene semantics from modality-specific appearance. By incorporating modality-aware contextual prompts during instruction tuning, VVM-Tuning enables zero-shot adaptation to novel visual modalities without requiring any target-modality training data. As the first systematic investigation into cross-modal generalization for LMMs, the study introduces VVM-Bench, a comprehensive evaluation benchmark encompassing six real and synthetic visual modalities. Experiments demonstrate consistent performance gains across five mainstream LMMs on all evaluated modalities, highlighting the effectiveness of the proposed approach in enhancing out-of-distribution visual understanding.

Abstract

Despite the advancements of Large Multimodal Models (LMMs) in RGB vision, their ability to generalize to unseen visual modalities remains a largely unexplored challenge. We argue that different visual modalities are merely distinct samplings of the same physical world. Therefore, effective generalization requires models to possess both modality-agnostic perception of scene semantics and the adaptability to modality-specific characteristics. To achieve this, we propose a training framework, VVM-Tuning, to equip LMMs with these capabilities through modality synthesis and modality contexts. Specifically, we synthesize diverse appearance-varied images from RGB scenes, training the model to disentangle invariant semantics from varying visual appearances, and align these appearances with language for visual concepts decoupled from modalities. We then introduce modality contexts in the prompt and use instruction tuning to assist the model in mapping these appearance variations back to modality-related attributes, enabling zero-shot adaptation to unseen modalities during inference. To facilitate research in this direction, we introduce VVM-Bench, a comprehensive benchmark featuring 6 real and synthetic modalities to evaluate semantic perception and modality understanding. Experiments demonstrate that, via our training on synthetic modalities, 5 tested models exhibit consistent improvements on both real-world and novel synthetic modalities without in-modality training. Source code and data will be publicly available at https://github.com/Hunter-Will/VVM-Tuning.

Problem

• Large Multimodal Models
• visual modalities
• generalization
• unseen modalities
• modality-agnostic perception

Innovation

• Modality Synthesis
• Zero-shot Generalization
• Modality-agnostic Perception
• Instruction Tuning
• Multimodal Benchmark