CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognition

2026-08-07

AI Summary

This work addresses the challenges of unreliable self-reported labels and overlooked cross-modal conflicts in multimodal emotion recognition by proposing a conflict-aware graph neural network framework. The approach models modality-specific experts as graph nodes and enables iterative evidence negotiation through uncertainty-aware compatibility assessment and reliability-guided asymmetric edge weights. Based on the degree of inter-modal conflict, samples are dynamically categorized into consensus, disagreement, or ambiguous groups, each subjected to tailored weakly supervised calibration strategies. The method introduces, for the first time, runtime reliability estimation and a conflict-driven supervision mechanism, significantly enhancing robustness against label noise and high-conflict instances. It achieves state-of-the-art leave-one-subject-out (LOSO) accuracy of 0.873 on AMIGOS-V and 0.854 on MAHNOB-V, outperforming existing approaches.

Abstract

Multimodal emotion recognition often treats self-reported labels as reliable supervision while overlooking self-report unreliability and cross-modal conflict. We propose \textbf{CONFER}, a graph-based conflict-aware evidence negotiation framework for weakly supervised multimodal emotion recognition. CONFER represents each modality expert as a node with a predictive belief, boundary-based uncertainty, and runtime reliability estimated from historical out-of-fold performance and current-sample uncertainty. Uncertainty-aware compatibility and reliability-directed asymmetric edge weights govern iterative message-passing negotiation, followed by peer-supported prediction readout. Conflict reduction, residual disagreement, and mean modality uncertainty further characterize three regimes---Consensus, Dissent, and Ambiguity---for sample-specific weak-label calibration. We evaluate CONFER on AMIGOS, MAHNOB-HCI, and DEAP under subject-dependent 10-fold and strict leave-one-subject-out (LOSO) protocols. CONFER achieves competitive performance, reaching \textbf{0.873} accuracy on AMIGOS-V and \textbf{0.854} accuracy on MAHNOB-V under strict LOSO evaluation. Further analyses show larger negotiation gains on high-conflict samples and improved robustness to weak-label corruption, indicating that cross-modal conflict provides useful information for both directional modality coordination and supervision-reliability estimation.

Problem

• multimodal emotion recognition
• weak supervision
• cross-modal conflict
• self-report unreliability
• label calibration

Innovation

• conflict-aware negotiation
• weak supervision calibration
• multimodal emotion recognition
• graph-based message passing
• uncertainty-aware reliability