Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR
This study addresses the high sensitivity of single-run evaluations to random seeds in low-resource Garhwali speech recognition, which often obscures genuine performance gains from stochastic noise. To remedy this, the authors establish the first reproducible ASR benchmark on the official VAANI dataset using multiple random seeds and propose a new evaluation paradigm centered on multi-seed assessment and statistical significance testing. Systematic re-evaluation of various optimization objectives and transfer strategies reveals that standard CTC combined with w2v-BERT 2.0 achieves a 47.0% WER across five seeds, outperforming larger models such as MMS-1B. While speed perturbation yields consistent minor improvements, more complex approaches like Focal CTC and matra weighting fail to demonstrate statistically significant gains. The findings underscore the fragility of common enhancements in low-resource settings and highlight the superiority of thoughtful pretraining design over mere model scale expansion.