Generative Testing of Automated Speech Recognition Systems

2026-07-10

AI Summary

This work addresses the vulnerability of black-box automatic speech recognition (ASR) systems to high-fidelity adversarial attacks, for which efficient methods to generate realistic failure-inducing samples remain scarce. The authors propose GATAS, a novel approach that—without access to model gradients—leverages phoneme-level latent space interpolation in text-to-speech models to synthesize natural-sounding speech. By integrating multi-objective optimization to balance semantic perturbation and perceptual quality, GATAS effectively induces transcription errors in target ASR systems. Experimental results demonstrate that GATAS achieves a 98% attack success rate while producing samples with lower distortion and superior subjective audio quality, matching the performance of white-box methods. These findings underscore the critical role of aligning representational fidelity with perceptual realism in effective black-box ASR testing.

Abstract

Automatic speech recognition (ASR) systems have achieved high accuracy with transformer-based models, enabling deployment in critical applications. However, they remain vulnerable to adversarial manipulation, particularly in black-box settings where attacks must preserve perceptual naturalness. This work introduces GATAS, a black-box testing approach that generates failure inducing inputs by operating in the phoneme-level latent space of a text- to-speech model. Instead of perturbing waveforms directly, the approach interpolates latent representations to induce transcription errors while remaining within the manifold of natural speech. The attack is formulated as a multi-objective optimization problem balancing semantic divergence and perceptual quality. Our empirical evaluation against both white-box and black-box baselines shows that GATAS achieves a 98% success rate while producing lower distortion and higher perceptual quality, as confirmed by human studies. Despite operating without gradient access, GATAS remains competitive against white-box methods, highlighting that representation and perceptual alignment are more critical than access to model internals. Overall, our results demonstrate that untargeted latent-space optimization enables the efficient generation of realistic and effective test cases for ASR systems.

Problem

• Automatic Speech Recognition
• Adversarial Testing
• Black-box Attack
• Perceptual Naturalness
• Robustness Evaluation

Innovation

• black-box testing
• latent-space optimization
• adversarial ASR
• phoneme-level interpolation
• perceptual naturalness