DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning

2026-04-20

AI Summary

This work addresses the limited robustness of existing multimodal large language models under visual degradations such as fog, blur, or low-light conditions. To this end, it presents the first systematic integration of infrared modality into multimodal large language models, introducing a lightweight patch-level local cross-attention fusion mechanism to efficiently combine RGB and infrared information. The authors construct DV-204K, the first large-scale aligned infrared-RGB visual question answering dataset, along with DV-500, a dedicated evaluation benchmark, to train and assess their approach. Experimental results demonstrate that the proposed method significantly enhances visual reasoning performance across diverse degradation scenarios, confirming the effectiveness of infrared-RGB fusion in improving model robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal large language models (MLLMs) have achieved impressive performance on visual perception and reasoning tasks with RGB imagery, yet they remain fragile under common degradations, such as fog, blur, or low-light conditions. Infrared (IR) imaging, a well-established complement to RGB, offers inherent robustness in these conditions, but its integration into MLLMs remains underexplored. To bridge this gap, we propose DUALVISION, a lightweight fusion module that efficiently incorporates IR-RGB information into MLLMs via patch-level localized cross-attention. To support training and evaluation and to facilitate future research, we also introduce DV-204K, a dataset of ~25K publicly available aligned IR-RGB image pairs with 204K modality-specific QA annotations, and DV-500, a benchmark of 500 IR-RGB image pairs with 500 QA pairs designed for evaluating cross-modal reasoning. Leveraging these datasets, we benchmark both open- and closed-source MLLMs and demonstrate that DUALVISION delivers strong empirical performance under a wide range of visual degradations. Our code and dataset are available at https://abrarmajeedi.github.io/dualvision.

Problem

• multimodal large language models
• RGB-infrared fusion
• visual degradation
• infrared imaging
• visual reasoning

Innovation

• multimodal large language models
• infrared-RGB fusion
• cross-modal reasoning
• visual robustness
• patch-level cross-attention