FlowEdit: Associative Memory for Lifelong Pronunciation Adaptation in Flow-Matching TTS

2026-06-18

AI Summary

This work addresses the challenge of dynamically correcting mispronunciations of out-of-vocabulary proper nouns in flow-matching text-to-speech (TTS) systems after deployment. To this end, we propose FlowEdit, a novel framework that introduces content-addressable memory into TTS pronunciation adaptation for the first time. Without fine-tuning or modifying the pre-trained model, FlowEdit enables rapid, fuzzy-match-based pronunciation updates through implicit conditioning in the text embedding space, leveraging modern Hopfield networks and soft-attention similarity gating. Experimental results on a multilingual test set of 312 proper nouns show a 92.7% relative reduction in phoneme error rate compared to the baseline, with each correction taking approximately 15 seconds on a single GPU, while preserving overall speech quality.

Abstract

Flow-matching text-to-speech systems achieve remarkable zero-shot quality but remain static after deployment: pronunciation errors on out-of-vocabulary proper nouns persist unless the model is retrained. We introduce FlowEdit, a life-long adaptation framework for frozen flow-matching TTS that learns pronunciation corrections as latent conditioning edits rather than weight updates. When corrective feedback is provided, FlowEdit optimizes a token-level perturbation in the text embedding space, then stores the correction in a Modern Hopfield Network serving as content-addressable episodic memory. At inference, corrections are retrieved via soft attention with a similarity gate, enabling fuzzy morphological matching. On our curated benchmark of 312 multilingual proper nouns across 18 language families, FlowEdit reduces target-word Phoneme Error Rate by 92.7% relative to the zero-shot baseline while maintaining identical general-speech quality. Corrections complete in approximately 15 seconds on a single GPU.

Problem

• pronunciation adaptation
• out-of-vocabulary
• proper nouns
• lifelong learning
• text-to-speech

Innovation

• FlowEdit
• lifelong adaptation
• associative memory
• flow-matching TTS
• pronunciation correction