Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras

2026-09-07

AI Summary

为解决传统RGB相机隐私风险问题，本文使用事件相机、音频和文本多模态数据，提出信息引导门控融合框架，提升复杂环境下情感识别性能。

Abstract

Emotion analysis is a fundamental task in computer vision, but its practical deployment remains constrained by the privacy risks inherent to conventional RGB cameras. Bio-inspired event cameras present a promising hardware-level solution because they capture asynchronous brightness changes, thereby reducing exposure of facial identity details while leveraging high dynamic range for robust perception under challenging illumination conditions. Despite these advantages, existing event-based methods struggle in complex real-world settings due to limited dataset scales, simple acquisition conditions, and reliance on single-modality visual cues. To address these, we establish a challenging tri-modal benchmark with event, audio, and text modalities and propose the Information-Guided Gated Fusion (IGF) framework, which first pre-trains an event encoder on the FAU subset of Emo-DVS to capture fine-grained facial dynamics, then employs adaptive modality gating to suppress modality-specific noise, and finally leverages mutual information maximization to align robust cross-modal representations. To alleviate data scarcity, we introduce Emo-DVS, the first large-scale event-based emotion analysis dataset, which couples dynamic illumination with the Facial Action Unit (FAU) subset and emotion subset. Extensive experiments demonstrate that IGF achieves state-of-the-art performance.

Problem

• Emotion Recognition
• Privacy-Aware
• Event Cameras
• Multimodal Benchmark

Innovation

• Information-Guided Gated Fusion
• multimodal emotion recognition
• event cameras
• adaptive modality gating
• cross-modal representation