LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion

2026-06-29

AI Summary

Personality recognition in asynchronous video interviews often suffers from insufficient information due to the neglect of subtle facial dynamics or overreliance on textual content alone. This work proposes a novel approach that transforms sequences of facial Action Units (AUs) into interpretable semantic descriptions, which are then semantically fused with interview transcripts within a large language model (LLM). A lightweight regression head subsequently predicts continuous personality scores. By introducing the first method to semantically interpret AUs and jointly model them with linguistic content in an LLM, this framework effectively integrates nonverbal and verbal cues while decoupling semantic understanding from regression, thereby enhancing both interpretability and training stability. Evaluated on the AVI-6 benchmark, the proposed method significantly outperforms existing baselines, achieving lower prediction error and higher correlation with human ratings, demonstrating its psychological plausibility and computational efficiency.

Abstract

Personality recognition in asynchronous video interviews (AVIs) has become increasingly important due to their widespread adoption in modern recruitment. Existing approaches often rely on large language models (LLMs) to analyze textual responses of interviewees in AVI. However, unimodel methods often suffer from information loss (e.g., ignore facial cues). In contrast, multimodal methods that employ full-face images or sparsely sampled frames can discard fine-grained temporal dynamics critical for accurate personality assessment. To overcome these limitations, we propose an LLM-based framework that semantically fuse facial action units (AUs) with textual responses of AVI. AU sequences are first converted into interpretable textual descriptions, which are then fused with participants' textual responses through an LLM. A lightweight regression head transforms the resulting embeddings into continuous personality scores without disrupting the underlying semantic space. Experiments on the AVI-6 benchmark demonstrate consistent improvements over most baselines, with lower prediction errors and stronger correlations with human-rated scores across multiple traits. Further analysis reveals that AU-derived semantic representations offer complementary non-verbal cues to textual responses. Decoupling semantic understanding from regression prediction within the LLM also leads to greater training stability and clearer interpretability. Overall, these findings demonstrate that AU-text fusion provides a psychologically grounded and computationally efficient framework for personality recognition in AVIs.

Problem

• personality recognition
• asynchronous video interviews
• multimodal fusion
• facial action units
• information loss

Innovation

• Facial Action Units
• Semantic Fusion
• Large Language Models
• Multimodal Personality Recognition
• Asynchronous Video Interviews