Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models

2026-08-27

AI Summary

研究探讨了多模态模型在处理语音和文本查询时的一致性问题，通过构建跨模态对比三元组基准并评估模型性能来解决。

Abstract

Multimodal foundation models are increasingly used in speech-first assistants that must interpret spoken queries and produce visually grounded decisions. Yet it remains unclear whether semantically equivalent queries yield consistent judgments across modality (text vs. speech) and language (English vs. Arabic). We introduce a speech-augmented visually grounded contrastive triplet benchmark spanning 10,150 culturally grounded images from 18 MENA countries, where each image is paired with one supported statement and two plausible but unsupported alternatives. We define contrastive instability as the conditional rate at which a model fails to resolve all statements within a triplet, isolating fragmented reasoning from complete failure. Evaluating recent multimodal models under text and speech in English and Arabic, we find that modality and language shifts introduce substantial triplet-level inconsistencies that are not fully captured by aggregate accuracy, with speech amplifying partial failures. We make the benchmark publicly available to the community.

Problem

• multimodal models
• cross-modal instability
• contrastive triplet

Innovation

• Cross-Modal Instability
• Multimodal Models
• Contrastive Triplet Benchmark