RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding

2026-05-19

AI Summary

This work addresses the performance limitations of existing vision-language models under challenging conditions—such as low light, high dynamic range, or rapid motion—where RGB image degradation severely impairs model efficacy. To overcome this, the paper introduces event camera data into vision-language modeling for the first time and proposes RE-VLM, the first dual-stream vision-language model that fuses RGB frames with event streams. The approach leverages a dual-stream encoding architecture, a heterogeneous feature alignment mechanism, and a progressive training strategy. Furthermore, it employs a graph-driven automatic synthesis method to generate large-scale RGB-event-text triplets, effectively mitigating the scarcity of multimodal annotations. Experiments demonstrate that RE-VLM significantly outperforms single-modality baselines on image captioning and visual question answering tasks over the PEOD-Chat and RGBE-Chat datasets, with particularly pronounced gains in challenging scenarios.

Abstract

Conventional vision-language models (VLMs) struggle to interpret scenes captured under adverse conditions (e.g., low light, high dynamic range, or fast motion) because standard RGB images degrade in such environments. Event cameras provide a complementary modality: they asynchronously record per-pixel brightness changes with high temporal resolution and wide dynamic range, preserving motion cues where frames fail. We propose RE-VLM, the first dual-stream vision-language model that jointly leverages RGB images and event streams for robust scene understanding across both normal and challenging conditions. RE-VLM employs parallel RGB and event encoders together with a progressive training strategy that aligns heterogeneous visual features with language. To address the scarcity of RGB-Event-Text supervision, we further propose a graph-driven pipeline that converts synchronized RGB-Event streams into verifiable scene graphs, from which we synthesize captions and question-answer (QA) pairs. To develop and evaluate RE-VLM, we construct two datasets: PEOD-Chat, targeting illumination-challenged scenes, and RGBE-Chat, covering diverse scenarios. On captioning and VQA benchmarks, RE-VLM consistently outperforms state-of-the-art RGB-only and event-only models with comparable parameter counts, with particularly large gains under challenging conditions. These results demonstrate the effectiveness of event-augmented VLMs in achieving robust vision-language understanding across a wide range of real-world environments. Code and datasets are available at https://github.com/bupt-ai-cz/RE-VLM.

Problem

• vision-language models
• adverse conditions
• event cameras
• scene understanding
• RGB degradation

Innovation

• event camera
• vision-language model
• dual-stream architecture
• scene graph generation
• adverse-condition perception