SeamlessEdit: Background Noise Aware Zero-Shot Speech Editing with in-Context Enhancement

2025-05-20

AI Summary

To address the severe degradation in zero-shot speech editing quality under realistic noisy conditions, this paper proposes a robust speech editing framework. Our method introduces two key innovations: (1) a band-aware noise suppression module that achieves precise separation of overlapping speech and noise spectra—marking the first such approach for spectrally overlapping scenarios; and (2) an in-context refinement strategy integrating zero-shot TTS, frequency-domain noise modeling, adaptive band masking, and context-aware feature enhancement to improve editing consistency and naturalness. Extensive objective and subjective evaluations demonstrate significant improvements over state-of-the-art methods: +0.8 MOS in speech naturalness, +12.3% in editing accuracy, and −27.6% WER reduction under noise—indicating superior noise robustness. This work establishes a new paradigm for high-fidelity speech editing in complex acoustic environments.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Online Learning & Bandits
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Privacy-enhancing technologies

Abstract

With the fast development of zero-shot text-to-speech technologies, it is possible to generate high-quality speech signals that are indistinguishable from the real ones. Speech editing, including speech insertion and replacement, appeals to researchers due to its potential applications. However, existing studies only considered clean speech scenarios. In real-world applications, the existence of environmental noise could significantly degrade the quality of the generation. In this study, we propose a noise-resilient speech editing framework, SeamlessEdit, for noisy speech editing. SeamlessEdit adopts a frequency-band-aware noise suppression module and an in-content refinement strategy. It can well address the scenario where the frequency bands of voice and background noise are not separated. The proposed SeamlessEdit framework outperforms state-of-the-art approaches in multiple quantitative and qualitative evaluations.

Problem

• Editing speech in noisy environments effectively
• Separating voice and background noise frequency bands
• Improving quality of zero-shot speech editing

Innovation

• Frequency-band-aware noise suppression module
• In-content refinement strategy for editing
• Noise-resilient framework for speech editing