Multi-modal expressive personality recognition in data non-ideal audiovisual based on multi-scale feature enhancement and modal augment

2025-03-08

AI Summary

This work addresses the insufficient robustness of explicit personality recognition under non-ideal audiovisual conditions—such as modality missingness and noise corruption. We propose an end-to-end audiovisual fusion network featuring a cross-modal attention mechanism for deep feature-level integration, augmented by a multi-scale feature enhancement module and a modality-augmented training strategy, significantly improving adaptability to incomplete or degraded data. On the ChaLearn First Impression dataset, our method achieves a mean Big Five personality recognition accuracy of 0.916—surpassing existing audiovisual multimodal approaches. Extensive evaluation across six non-ideal scenarios demonstrates superior robustness and generalizability. Our core contributions are threefold: (i) the first systematic formulation of multimodal personality recognition under non-ideal audiovisual conditions; (ii) a structured fusion architecture enabling precise expressive modeling; and (iii) a collaborative enhancement paradigm that jointly optimizes modeling fidelity and real-world deployment reliability.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Automatic personality recognition is a research hotspot in the intersection of computer science and psychology, and in human-computer interaction, personalised has a wide range of applications services and other scenarios. In this paper, an end-to-end multimodal performance personality is established for both visual and auditory modal datarecognition network , and the through feature-level fusion , which effectively of the two modalities is carried out the cross-attention mechanismfuses the features of the two modal data; and a is proposed multiscale feature enhancement modalitiesmodule , which enhances for visual and auditory boththe expression of the information of effective the features and suppresses the interference of the redundant information. In addition, during the training process, this paper proposes a modal enhancement training strategy to simulate non-ideal such as modal loss and noise interferencedata situations , which enhances the adaptability ofand the model to non-ideal data scenarios improves the robustness of the model. Experimental results show that the method proposed in this paper is able to achieve an average Big Five personality accuracy of , which outperforms existing 0.916 on the personality analysis dataset ChaLearn First Impressionother methods based on audiovisual and audio-visual both modalities. The ablation experiments also validate our proposed , respectivelythe contribution of module and modality enhancement strategy to the model performance. Finally, we simulate in the inference phase multi-scale feature enhancement six non-ideal data scenarios to verify the modal enhancement strategy's improvement in model robustness.

Problem

• Enhances multi-modal personality recognition using feature-level fusion and cross-attention mechanisms.
• Proposes a multi-scale feature enhancement module to improve visual and auditory data representation.
• Introduces a modal enhancement strategy to improve model robustness in non-ideal data scenarios.

Innovation

• End-to-end multimodal network for personality recognition
• Multiscale feature enhancement for visual and auditory data
• Modal enhancement training for non-ideal data robustness