LABNet: A Lightweight Attentive Beamforming Network for Ad-hoc Multichannel Microphone Invariant Real-Time Speech Enhancement

2025-07-21

AI Summary

This work addresses real-time multi-channel speech enhancement under unconstrained microphone arrays—where the number and geometric configuration of microphones are variable. We propose a lightweight attention-based beamforming network that employs a three-stage feature extraction and fusion framework, incorporating cross-channel attention to achieve microphone-invariant modeling for the first time at extremely low computational complexity. The method enables adaptive feature aggregation across arbitrary numbers and configurations of microphones. By integrating time-frequency domain beamforming with a compact neural architecture, it achieves real-time inference on edge devices. Experiments demonstrate that the model achieves superior speech quality and intelligibility over existing lightweight approaches, while reducing parameters (<1M) and computational cost (<0.5 GFLOPs). Crucially, it maintains robust performance across diverse array geometries, establishing a new paradigm for on-device speech enhancement in unconstrained array scenarios.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Speech
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Multichannel speech enhancement (SE) aims to restore clean speech from noisy measurements by leveraging spatiotemporal signal features. In ad-hoc array conditions, microphone invariance (MI) requires systems to handle different microphone numbers and array geometries. From a practical perspective, multichannel recordings inevitably increase the computational burden for edge-device applications, highlighting the necessity of lightweight and efficient deployments. In this work, we propose a lightweight attentive beamforming network (LABNet) to integrate MI in a low-complexity real-time SE system. We design a three-stage framework for efficient intra-channel modeling and inter-channel interaction. A cross-channel attention module is developed to aggregate features from each channel selectively. Experimental results demonstrate our LABNet achieves impressive performance with ultra-light resource overhead while maintaining the MI, indicating great potential for ad-hoc array processing.

Problem

• Handles varying microphone numbers and geometries in ad-hoc arrays
• Reduces computational burden for real-time edge-device speech enhancement
• Achieves lightweight deployment with microphone invariance and efficiency

Innovation

• Lightweight attentive beamforming network for SE
• Three-stage framework for efficient channel modeling
• Cross-channel attention module for selective feature aggregation