"Alexa, can you forget me?"Machine Unlearning Benchmark in Spoken Language Understanding

2025-05-21

AI Summary

This paper addresses speaker-level data removal under the “right to be forgotten” for spoken language understanding (SLU). To this end, we introduce UnSLU-BENCH—the first SLU-specific machine unlearning benchmark—covering four languages and four SLU datasets, along with a multi-dimensional evaluation framework balancing effectiveness, efficiency, and practicality. We systematically evaluate eight categories of unlearning methods, including ensemble editing, influence functions, retraining, and knowledge distillation, integrated with speaker de-identification, semantic consistency verification, and joint ASR-SLU assessment. Experiments reveal substantial trade-offs among unlearning quality, model utility retention, and computational overhead, establishing foundational insights for SLU unlearning research. The benchmark is publicly released to support reproducible, standardized evaluation of privacy-compliant speech AI systems.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Privacy
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Machine unlearning, the process of efficiently removing specific information from machine learning models, is a growing area of interest for responsible AI. However, few studies have explored the effectiveness of unlearning methods on complex tasks, particularly speech-related ones. This paper introduces UnSLU-BENCH, the first benchmark for machine unlearning in spoken language understanding (SLU), focusing on four datasets spanning four languages. We address the unlearning of data from specific speakers as a way to evaluate the quality of potential"right to be forgotten"requests. We assess eight unlearning techniques and propose a novel metric to simultaneously better capture their efficacy, utility, and efficiency. UnSLU-BENCH sets a foundation for unlearning in SLU and reveals significant differences in the effectiveness and computational feasibility of various techniques.

Problem

• Evaluating machine unlearning effectiveness in spoken language understanding tasks
• Assessing unlearning techniques for speaker-specific data removal in SLU
• Introducing a benchmark and metric for unlearning quality and efficiency

Innovation

• Introduces UnSLU-BENCH for SLU unlearning benchmark
• Evaluates eight unlearning techniques for speaker data removal
• Proposes novel metric for efficacy, utility, and efficiency