VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness

2026-07-09

AI Summary

This study addresses the scarcity of large-scale visual speech recognition (VSR) datasets for Romanian by introducing VSRo-200, the first dataset comprising 200 hours of authentic podcast videos, annotated through a combination of manual transcription and pseudo-labels generated by a fine-tuned automatic speech recognition (ASR) model. Leveraging this resource, the authors systematically evaluate the impact of supervision quality, out-of-domain generalization, and audio-visual fusion under low-resource conditions. Experimental results demonstrate that pseudo-labeling improves recognition performance as data scale increases, while multimodal fusion substantially enhances robustness to noise and significantly outperforms existing approaches on the LRRo isolated-word recognition task. This work establishes a critical benchmark and foundational dataset for Romanian visual speech recognition research.

Abstract

We introduce VSRo-200, the first large-scale dataset for visual speech recognition (lip reading) in Romanian, comprising 200 hours of real-world podcast videos. All samples are annotated with pseudo-labels generated by a fine-tuned Romanian ASR model, while a subset of 100 hours is additionally transcribed by humans, enabling controlled analysis of supervision quality under a unified framework. Building on this dataset, we establish a benchmark for visual speech recognition in low-resource settings. We systematically study the impact of supervision quality, showing that while human annotations provide better performance at fixed data scales, pseudo-labels enable continued improvements through scalability. We further evaluate robustness under domain shift using curated out-of-distribution (OOD) test sets, and analyze audio-visual speech recognition (AVSR) under noisy conditions, where multimodal fusion significantly improves robustness compared to audio-only models. Finally, we demonstrate that representations learned on VSRo-200 transfer effectively to the LRRo benchmark for isolated word recognition, substantially outperforming previously reported results. Overall, VSRo-200 provides a new testbed for studying supervision, domain generalization, and multimodal fusion in low-resource visual speech recognition.

Problem

• visual speech recognition
• low-resource
• supervision quality
• domain generalization
• multimodal robustness

Innovation

• visual speech recognition
• pseudo-labeling
• multimodal fusion
• domain generalization
• low-resource