SIGMA: Saliency-Guided Sparse Mask Attacks for Speech Emotion Recognition

2026-06-29

AI Summary

Current sparse adversarial attacks on speech emotion recognition lack interpretability guidance and metrics for explanation consistency, while struggling to simultaneously satisfy sparsity, perturbation magnitude constraints, and cross-model transferability. This work proposes SIGMA, a saliency-guided sparse masking attack that, for the first time, leverages post-hoc explainable AI (XAI)-generated saliency maps to direct structured perturbations on self-supervised speech representations. By computing a saliency mask in a single pass to restrict perturbation regions and applying magnitude-constrained optimized perturbations within those regions, SIGMA achieves attack success rates comparable to existing methods on the IEMOCAP and TESS datasets, while significantly improving explanation consistency and cross-model transferability.

Abstract

Speech conveys rich emotional information. As Speech Emotion Recognition (SER) is usually deployed in privacy-sensitive and reliability-critical environments, adversarial attacks on SER have attracted increasing attention. Existing sparse attacks control the number of perturbed elements, yet, they often lack explainability guidance and explicit measures of explanation consistency. A unified treatment of sparsity and magnitude constraints is also uncommon. In addition, transferability across attack families and target models remains limited. Hence, we propose a SalIency-Guided sparse Mask Attack (SIGMA). On self-supervised speech features, we use post-hoc explainable artificial intelligence (XAI) techniques to produce saliency maps and identify the scope of the mask, and then restrict magnitude-bounded updates to this mask. The mask is computed once and can be reused across models and different sparsity attacks to amortise cost. We evaluate on the IEMOCAP and TESS datasets. Under matched budgets and across multiple sparse-attack settings, SIGMA maintains competitive attack success rates, navigating a conscious trade-off between attack efficacy and explanation consistency. SIGMA therefore provides an efficient and interpretable framework for analysing the vulnerability and explanation behaviour of SER models under structured perturbations.

Problem

• adversarial attacks
• speech emotion recognition
• sparse attacks
• explainability
• transferability

Innovation

• saliency-guided
• sparse mask attack
• explainable AI
• adversarial robustness
• speech emotion recognition