🤖 AI Summary
This work addresses the privacy risks and manual effort inherent in traditional interview-based emotion analysis by proposing the first fully on-device, end-to-end emotion-driven conversational analysis framework. The system integrates speaker diarization, Whisper-based automatic speech recognition (ASR), and a wav2vec2-based emotion classifier to generate time-stamped emotional evidence—all processed locally without internet connectivity. A tri-model ensemble of local large language models then performs citation-constrained question-answering over this evidence. Evaluated on four subsets of RAVDESS, the approach achieves 48.8% emotion classification accuracy, significantly outperforming baseline methods. The entire pipeline runs on CPU with an average latency of 157 seconds (real-time factor 1.33), offering strong privacy guarantees, auditability, and cross-corpus emotional evidence integration, while candidly acknowledging its transfer limitations and the necessity of human validation boundaries.
📝 Abstract
Reviewing recorded interviews for affective cues such as composure, hesitation and agitation is slow and subjective, and cloud services that could automate it require sensitive audio to leave the device. EmotionAI is a fully local Computational Intelligence (CI) pipeline that couples Speech Emotion Recognition (SER) with generative reasoning. Speaker diarisation, Whisper Automatic Speech Recognition (ASR) and a wav2vec2 emotion classifier produce per-segment affective evidence, which is then passed to an adversarial three-model local Large Language Model (LLM) panel for timestamp-grounded and citation-constrained question answering. Zero-shot evaluation on the RAVDESS four-class English subset (n = 672) exposes cross-corpus fragility rather than classifier superiority: the deployed classifier scores 48.8% accuracy, above random (24.9%) and majority (28.6%) baselines but below an in-domain MFCC + logistic-regression comparator (71.0%). The complete pipeline runs in a mean 157 s on CPU (real-time factor approximately 1.33) with zero external calls. The contribution is not state-of-the-art SER but an auditable, privacy-preserving integration of imperfect affective evidence into grounded conversational analysis, together with an honest empirical account of where cross-corpus transfer and human-centred validation still fall short.