🤖 AI Summary
This study addresses the challenge of inefficient multimodal (visual-auditory) module coordination in assistive perception systems. We propose a lightweight, domain-specific modular deep learning framework: a CNN processes eye-region images for gaze-state estimation; a deeper CNN models facial expressions (trained on FER2013); and a CNN-LSTM hybrid architecture performs speaker identification (using a custom audio dataset). Each module is independently optimized and designed for plug-and-play integration. Our key contribution is empirically validating that high-accuracy unimodal modeling combined with a loosely coupled modular architecture achieves superior performance and deployment flexibility under resource constraints. Experiments yield accuracies of 93.0% (gaze state), 97.8% (facial expression), and 96.89% (speaker ID), significantly outperforming end-to-end joint modeling baselines. This work establishes a scalable, modular paradigm for assistive technologies.
📝 Abstract
Developing comprehensive assistive technologies requires the seamless integration of visual and auditory perception. This research evaluates the feasibility of a modular architecture inspired by core functionalities of perceptive systems like'Smart Eye.'We propose and benchmark three independent sensing modules: a Convolutional Neural Network (CNN) for eye state detection (drowsiness/attention), a deep CNN for facial expression recognition, and a Long Short-Term Memory (LSTM) network for voice-based speaker identification. Utilizing the Eyes Image, FER2013, and customized audio datasets, our models achieved accuracies of 93.0%, 97.8%, and 96.89%, respectively. This study demonstrates that lightweight, domain-specific models can achieve high fidelity on discrete tasks, establishing a validated foundation for future real-time, multimodal integration in resource-constrained assistive devices.