FeatureSense: Protecting Speaker Attributes in Always-On Audio Sensing System

2025-05-30

AI Summary

In always-on audio sensing systems, speaker-sensitive attributes (e.g., age, gender) remain vulnerable to inference even after voice masking, compromising user privacy. Method: We propose a privacy-preserving yet utility-aware solution: (1) the first systematic framework for quantifying speaker attribute leakage; (2) a generalizable privacy-aware audio feature library coupled with an adaptive, task-driven feature selection algorithm to jointly optimize privacy protection, recognition accuracy, and computational efficiency; and (3) integration of differential-privacy-inspired feature perturbation with robust feature modeling. Contribution/Results: Experiments across diverse audio sensing tasks demonstrate sustained high recognition accuracy while reducing speaker attribute leakage by 60.6% over state-of-the-art methods—enabling practical deployment of trustworthy audio classification systems.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: AI for Accessibility

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Privacy-enhancing technologies
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Audio is a rich sensing modality that is useful for a variety of human activity recognition tasks. However, the ubiquitous nature of smartphones and smart speakers with always-on microphones has led to numerous privacy concerns and a lack of trust in deploying these audio-based sensing systems. This paper addresses this critical challenge of preserving user privacy when using audio for sensing applications while maintaining utility. While prior work focuses primarily on protecting recoverable speech content, we show that sensitive speaker-specific attributes such as age and gender can still be inferred after masking speech and propose a comprehensive privacy evaluation framework to assess this speaker attribute leakage. We design and implement FeatureSense, an open-source library that provides a set of generalizable privacy-aware audio features that can be used for wide range of sensing applications. We present an adaptive task-specific feature selection algorithm that optimizes the privacy-utility-cost trade-off based on the application requirements. Through our extensive evaluation, we demonstrate the high utility of FeatureSense across a diverse set of sensing tasks. Our system outperforms existing privacy techniques by 60.6% in preserving user-specific privacy. This work provides a foundational framework for ensuring trust in audio sensing by enabling effective privacy-aware audio classification systems.

Problem

• Protecting speaker attributes in audio sensing systems
• Balancing privacy and utility in audio applications
• Preventing leakage of sensitive speaker-specific information

Innovation

• Open-source library for privacy-aware audio features
• Adaptive task-specific feature selection algorithm
• Comprehensive privacy evaluation framework