🤖 AI Summary
This study addresses the limited robustness of current event detection systems in real-world noisy scenarios for low-resource languages such as Bengali, despite their strong performance on clean text. The authors construct a Bengali news event benchmark dataset comprising 9,979 annotated sentences spanning clean text, ASR transcripts, and spelling-perturbed variants. They systematically evaluate the noise robustness of encoder-based models (e.g., BanglaBERT, XLM-R) and instruction-tuned decoder large language models (e.g., Llama 3, Gemma 2). The work reveals a fundamental difference in noise resilience between the two architectures: decoders exhibit greater robustness when trigger words are corrupted. By integrating annotation guidelines into instruction tuning and training on mixed clean-noisy data, the performance gap is substantially narrowed. Combining model scaling with multi-source joint training yields state-of-the-art results across diverse noise conditions.
📝 Abstract
Event detection (ED) systems are typically evaluated on clean, curated text, leaving their robustness to real-world noise largely unexplored, particularly for low-resource languages such as Bangla. We introduce a generalized Bangla news event ontology and a benchmark comprising 9,979 annotated sentences across 40 event subtypes, spanning clean news text, real-world Automatic Speech Recognition (ASR) transcripts, and orthographically corrupted text. We systematically evaluate fine-tuned encoder-only models (BanglaBERT and XLM-R) alongside instruction-tuned decoder-only large language models (Llama 3 and Gemma 3). Our results reveal a clear architectural trade-off: encoder models achieve higher performance on clean text but degrade substantially under noise, whereas decoder-only LLMs are markedly more robust, particularly when event triggers are corrupted. We further show that embedding annotation guidelines during instruction tuning establishes a higher performance baseline on noisy text but yields inconsistent reductions in performance degradation across noisy conditions. Finally, model scaling consistently improves the robustness of decoder-only LLMs, while combined training on clean and noisy data serves as an effective regularization strategy that disproportionately benefits encoder architectures, significantly narrowing the robustness gap.