"I've Heard of You!": Generate Spoken Named Entity Recognition Data for Unseen Entities

2024-12-26

AI Summary

Speech-based named entity recognition (NER) suffers from poor generalization to out-of-vocabulary (OOV) person and location names, while manual annotation of speech data is prohibitively expensive. Method: We propose a Named Entity Dictionary (NED)-driven synthetic data generation framework: leveraging large language models (LLMs) to generate text sentences containing novel entities, converting them to speech via text-to-speech (TTS), and applying noise-aware quality filtering. Contribution/Results: We introduce the first zero-shot evaluation benchmark for speech NER targeting unseen entities; propose an LLM+TTS co-generation paradigm with noise-perceptive filtering for low-cost, high-fidelity synthetic speech data; and achieve state-of-the-art performance across in-domain, zero-shot domain adaptation, and fully zero-shot settings. We publicly release (i) an NED containing 8,853 entities, (ii) the new benchmark dataset, and (iii) all source code.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Spoken named entity recognition (NER) aims to identify named entities from speech, playing an important role in speech processing. New named entities appear every day, however, annotating their Spoken NER data is costly. In this paper, we demonstrate that existing Spoken NER systems perform poorly when dealing with previously unseen named entities. To tackle this challenge, we propose a method for generating Spoken NER data based on a named entity dictionary (NED) to reduce costs. Specifically, we first use a large language model (LLM) to generate sentences from the sampled named entities and then use a text-to-speech (TTS) system to generate the speech. Furthermore, we introduce a noise metric to filter out noisy data. To evaluate our approach, we release a novel Spoken NER benchmark along with a corresponding NED containing 8,853 entities. Experiment results show that our method achieves state-of-the-art (SOTA) performance in the in-domain, zero-shot domain adaptation, and fully zero-shot settings. Our data will be available at https://github.com/DeepLearnXMU/HeardU.

Problem

• Novel Name Recognition
• Oral NER System
• Data Annotation Cost

Innovation

• Economic NER Data Generation
• TTS Integration for NER
• Quality Control in Data Generation