Personalized Keyword Spotting for User-Defined Keywords Leveraging Text-Independent Speaker Verification

2026-06-18

AI Summary

This work addresses the challenge of user-defined keyword spotting under dual zero-shot conditions—unseen keywords and unseen speakers—where existing systems struggle to reject impostors effectively. To this end, we propose ZP-KWS, a lightweight framework that integrates a phoneme-supervised audio encoder with a compact speaker encoder pretrained using generalized end-to-end (GE2E) loss. Keyword detection and speaker verification are jointly performed via multiplicative late fusion. A novel dual-branch architecture enables flexible switching between standard detection and strict speaker-gated modes without retraining, as each branch can be independently enabled or disabled. With only 1.55 million parameters, ZP-KWS is suitable for edge deployment and achieves up to a 60% relative reduction in false rejection rate (FRR) at 1% false acceptance rate (FAR) for target speakers on LibriPhrase, Google Speech Commands, and Qualcomm datasets, while maintaining strong keyword detection performance.

Abstract

User-defined keyword spotting (UD-KWS) enables zero-shot wake-word detection from text, but existing systems learn speaker-invariant representations that cannot reject impostors uttering the correct keyword. We address this dual zero-shot setting -- unseen keywords and unseen speakers -- with ZP-KWS, a lightweight framework combining a phoneme-supervised audio encoder with a GE2E-pretrained compact speaker encoder (about 0.9M parameters). Multiplicative late fusion at inference grants each branch independent veto power, supporting modes from conventional detection to strict speaker-gated activation without retraining. On LibriPhrase, Google Speech Commands, and Qualcomm datasets, ZP-KWS reduces target-only FRR at 1% FAR by up to 60% relative to the strongest baseline while maintaining competitive keyword detection, all within a 1.55M parameter budget for edge deployment.

Problem

• keyword spotting
• speaker verification
• zero-shot learning
• impostor rejection
• user-defined keywords

Innovation

• zero-shot keyword spotting
• speaker verification
• multiplicative late fusion
• lightweight framework
• user-defined wake-word