GRM: Utility-Aware Jailbreak Attacks on Audio LLMs via Gradient-Ratio Masking

2026-04-10

AI Summary

This work addresses the trade-off between attack efficacy and utility preservation in jailbreaking audio large language models (Audio-LLMs), where existing methods often degrade speech transcription quality and question-answering performance. To overcome this limitation, the authors propose a frequency-selective jailbreaking framework that leverages a gradient ratio masking mechanism to identify critical Mel-frequency bands that contribute significantly to successful attacks yet exhibit low sensitivity to utility degradation. By perturbing only these selected bands, the method generates reusable, universal adversarial perturbations. Evaluated across four mainstream Audio-LLMs, the approach achieves an average jailbreak success rate of 88.46%, substantially outperforming baseline methods while effectively preserving semantic intelligibility and task usability of the audio inputs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements

Abstract

Audio large language models (ALLMs) enable rich speech-text interaction, but they also introduce jailbreak vulnerabilities in the audio modality. Existing audio jailbreak methods mainly optimize jailbreak success while overlooking utility preservation, as reflected in transcription quality and question answering performance. In practice, stronger attacks often come at the cost of degraded utility. To study this trade-off, we revisit existing attacks by varying their perturbation coverage in the frequency domain, from partial-band to full-band, and find that broader frequency coverage does not necessarily improve jailbreak performance, while utility consistently deteriorates. This suggests that concentrating perturbation on a subset of bands can yield a better attack-utility trade-off than indiscriminate full-band coverage. Based on this insight, we propose GRM, a utility-aware frequency-selective jailbreak framework. It ranks Mel bands by their attack contribution relative to utility sensitivity, perturbs only a selected subset of bands, and learns a reusable universal perturbation under a semantic-preservation objective. Experiments on four representative ALLMs show that GRM achieves an average Jailbreak Success Rate (JSR) of 88.46% while providing a better attack-utility trade-off than representative baselines. These results highlight the potential of frequency-selective perturbation for better balancing attack effectiveness and utility preservation in audio jailbreak. Content Warning: This paper includes harmful query examples and unsafe model responses.

Problem

• audio jailbreak
• utility preservation
• attack-utility trade-off
• audio large language models
• frequency-selective perturbation

Innovation

• gradient-ratio masking
• utility-aware attack
• frequency-selective perturbation
• audio jailbreak
• universal adversarial perturbation