MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment

2026-08-11

AI Summary

Existing vision-language models rely on global image-text alignment, which struggles to establish fine-grained object-entity correspondences, leading to referential ambiguity and insufficient semantic grounding. This work proposes a multimodal code-switching (MMCS) pretraining paradigm, introducing the linguistic concept of code-switching into multimodal learning for the first time. By explicitly replacing textual entities with their corresponding visual objects, MMCS achieves object-level vision-language alignment. We construct a synthetic dataset comprising 773K samples and design a scalable data synthesis pipeline. Experiments demonstrate that an MMCS model trained on merely 50K samples matches or surpasses baseline models trained on 600K image-text pairs, significantly enhancing visual grounding and perception across various model scales while substantially improving data efficiency and semantic alignment accuracy.

Abstract

Existing Multimodal Large Language Models (MLLMs) predominantly rely on image-text pairs for modality alignment pretraining, mapping global image representations to long textual descriptions. However, this image-level alignment suffers from referential ambiguity: models struggle to infer the correspondences between multiple visual objects and textual entities from the global representation, leading to data inefficiency and suboptimal semantic grounding. To address this, we propose MultiModal Code-Switching (MMCS), a novel pretraining paradigm that provides explicit object-level supervision. Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding. We further develop a scalable data synthesis pipeline to generate a pretraining dataset of 773K samples with accurate object-entity correspondences. Experiments show that MMCS is highly data-efficient: with only 50K samples, it matches or surpasses models trained on 600K image-text pairs. Furthermore, MMCS consistently improves visual grounding and perception capabilities across varying model scales.

Problem

• referential ambiguity
• object-level alignment
• multimodal grounding
• vision-language correspondence
• semantic grounding

Innovation

• MultiModal Code-Switching
• object-level alignment
• visual grounding
• data-efficient pretraining
• vision-language grounding