MaskedFOP: Polyglot Speaker Identification under Missing Visual Modality via Cascaded Graph Label Propagation

2026-06-12

AI Summary

This work addresses the challenging scenario of multilingual speaker identification where facial modality is entirely absent during testing and speech is in an unseen language (Urdu). The proposed approach integrates a modality-drop dual-head network, multimodal embedding fusion, and a two-stage cascaded graph label propagation (GLP) framework. Audio features are extracted using FOP backbone and ECAPA-TDNN architectures, with modality dropout incorporated during training to enhance robustness. A novel cascaded seed augmentation mechanism effectively expands sparse training prototypes to thousands of test centers, substantially improving cross-lingual generalization. The method achieved state-of-the-art performance in the POLY-SIM 2026 challenge, securing first place with an average P-accuracy of 0.9989. Ablation studies confirm that the cascaded seeding strategy alone contributes over an 8-percentage-point performance gain.

Abstract

We present MaskedFOP, a system for closed-set polyglot speaker identification under two simultaneous challenges: the face modality is entirely absent at test time, and speech comes from Urdu, a language unseen during face-supervised training. The system integrates three complementary mechanisms. First, a modality-dropout dual-head network built on the Fusion and Orthogonal Projection (FOP) backbone forces the audio branch to develop independent discriminative power via per-sample face masking, ensuring that the audio encoder remains capable when face is absent. Second, two MaskedFOP instances trained on Emphasized Channel Attention, Propagation, and Aggregation in Time Delay Neural Network (ECAPA-TDNN) features with different random seeds produce complementary audio embeddings whose element-wise average yields a more robust 512-dimensional representation than any single model. Third, a two-stage cascaded inference procedure first refines multimodal labels through a fused Graph Label Propagation (GLP) pass (Stage 1), then assigns audio-only labels by cosine nearest-centroid (Stage 2), replacing the 70 sparse training prototypes with ~1,500 in-domain test-set centroids from Stage 1. Submitted to the POLY-SIM 2026 Grand Challenge, the system achieves a mean P-accuracy of 0.9989, placing first among all submissions evaluated on the challenge server. An ablation identifies cascaded seeding as the single largest gain (>8 pp on P4/P6). The code is available at https://github.com/Ayoub-Elkhouzari/POLY-SIM2026.

Problem

• polyglot speaker identification
• missing visual modality
• unseen language
• audio-only recognition
• closed-set identification

Innovation

• MaskedFOP
• modality dropout
• graph label propagation
• polyglot speaker identification
• cascaded inference