Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

2026-03-11

AI Summary

It remains unclear whether current speech-aware large language models (LLMs) possess speaker identification capabilities, and effective evaluation and enhancement methods are lacking. This work presents the first systematic assessment of their speaker discrimination ability and introduces a model-agnostic scoring protocol. Furthermore, it proposes a lightweight enhancement strategy that injects frozen ECAPA-TDNN speaker embeddings into the LLM and trains only a LoRA adapter, leaving the LLM backbone untouched. This approach preserves the natural language interface while achieving an equal error rate (EER) of 1.03% on VoxCeleb1-E—significantly outperforming the original speech-aware LLM (EER > 20%) and approaching the performance of dedicated speaker verification systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Speech-aware large language models (LLMs) can accept speech inputs, yet their training objectives largely emphasize linguistic content or specific fields such as emotions or the speaker's gender, leaving it unclear whether they encode speaker identity. First, we propose a model-agnostic scoring protocol that produces continuous verification scores for both API-only and open-weight models, using confidence scores or log-likelihood ratios from the Yes/No token probabilities. Using this protocol, we benchmark recent speech-aware LLMs and observe weak speaker discrimination (EERs above 20% on VoxCeleb1). Second, we introduce a lightweight augmentation that equips an LLM with ASV capability by injecting frozen ECAPA-TDNN speaker embeddings through a learned projection and training only LoRA adapters. On TinyLLaMA-1.1B, the resulting ECAPA-LLM achieves 1.03% EER on VoxCeleb1-E, approaching a dedicated speaker verification system while preserving a natural-language interface.

Problem

• speaker verification
• speech-aware LLMs
• speaker identity
• large language models
• voice biometrics

Innovation

• speech-aware LLMs
• speaker verification
• model-agnostic scoring
• ECAPA-TDNN
• LoRA adaptation