Problem
Research questions and friction points this paper is trying to address.
low-resource ASR
Garhwali
reproducibility
multi-seed evaluation
evaluation bias
Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR
🤖 AI Summary
This study addresses the high sensitivity of single-run evaluations to random seeds in low-resource Garhwali speech recognition, which often obscures genuine performance gains from stochastic noise. To remedy this, the authors establish the first reproducible ASR benchmark on the official VAANI dataset using multiple random seeds and propose a new evaluation paradigm centered on multi-seed assessment and statistical significance testing. Systematic re-evaluation of various optimization objectives and transfer strategies reveals that standard CTC combined with w2v-BERT 2.0 achieves a 47.0% WER across five seeds, outperforming larger models such as MMS-1B. While speed perturbation yields consistent minor improvements, more complex approaches like Focal CTC and matra weighting fail to demonstrate statistically significant gains. The findings underscore the fragility of common enhancements in low-resource settings and highlight the superiority of thoughtful pretraining design over mere model scale expansion.
📝 Abstract
At corpus sizes typical of low-resource dialects, single-run comparisons can yield gains that do not replicate. We show this for Garhwali, an under-resourced Indo-Aryan language of the central Himalaya, building the first reproducible multi-seed ASR benchmark on the official VAANI splits, with per-seed outputs and significance testing. Re-examining plausible gains, we find them fragile: neither Focal CTC nor a matra-weighted objective beats standard CTC under seed-level testing, the matra objective fails to cut even its targeted errors, and Hindi-to-Garhwali transfer gives no gain over direct fine-tuning. What holds up is mundane: w2v-BERT 2.0 with standard CTC reaches 47.0% WER over five seeds, beating the larger MMS-1B and comparable models; pretraining design, not parameter count, drives performance, and speed augmentation gives a small, largely consistent gain. Multi-seed evaluation on official splits separates real gains from seed noise.
Innovation
Methods, ideas, or system contributions that make the work stand out.
multi-seed evaluation
low-resource ASR
reproducible benchmark
CTC robustness
w2v-BERT 2.0
🔎 Similar Papers
No similar papers found.
