NeuroLip: An Event-driven Spatiotemporal Learning Framework for Cross-Scene Lip-Motion-based Visual Speaker Recognition

2026-04-17

AI Summary

This work addresses the limited generalization of conventional frame-based visual speaker recognition under cross-view and varying illumination conditions, as well as its inability to capture fine-grained lip motion dynamics. The authors propose the first event-camera-based spatiotemporal learning framework, which achieves robust generalization to unseen viewpoints and low-light scenarios after training under a single controlled condition. The method integrates a time-aware voxel encoding scheme, a structure-aware spatial enhancer, and polarity consistency regularization to effectively preserve directional lip motion cues while suppressing noise. Evaluated on the newly introduced DVSpeaker dataset, the approach attains cross-view recognition accuracy exceeding 71% and nearly 76% under low-light conditions, outperforming existing methods by at least 8.54%.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Visual speaker recognition based on lip motion offers a silent, hands-free, and behavior-driven biometric solution that remains effective even when acoustic cues are unavailable. Compared to traditional methods that rely heavily on appearance-dependent representations, lip motion encodes subject-specific behavioral dynamics driven by consistent articulation patterns and muscle coordination, offering inherent stability across environmental changes. However, capturing these robust, fine-grained dynamics is challenging for conventional frame-based cameras due to motion blur and low dynamic range. To exploit the intrinsic stability of lip motion and address these sensing limitations, we propose NeuroLip, an event-based framework that captures fine-grained lip dynamics under a strict yet practical cross-scene protocol: training is performed under a single controlled condition, while recognition must generalize to unseen viewing and lighting conditions. NeuroLip features a 1) Temporal-aware Voxel Encoding module with adaptive event weighting, 2) Structure-aware Spatial Enhancer that amplifies discriminative behavioral patterns by suppressing noise while preserving vertically structured motion information, and 3) Polarity Consistency Regularization mechanism to retain motion-direction cues encoded in event polarities. To facilitate systematic evaluation, we introduce DVSpeaker, a comprehensive event-based lip-motion dataset comprising 50 subjects recorded under four distinct viewpoint and illumination scenarios. Extensive experiments demonstrate that NeuroLip achieves near-perfect matched-scene accuracy and robust cross-scene generalization, attaining over 71% accuracy on unseen viewpoints and nearly 76% under low-light conditions, outperforming representative existing methods by at least 8.54%. The dataset and code are publicly available at https://github.com/JiuZeongit/NeuroLip.

Problem

• visual speaker recognition
• lip motion
• cross-scene generalization
• event-based vision
• biometric recognition

Innovation

• event-based vision
• lip-motion dynamics
• cross-scene generalization
• temporal-aware encoding
• polarity consistency regularization