🤖 AI Summary
Existing stemmer evaluation methods fail to quantify the semantic degradation caused by over-stemming in downstream tasks. This paper proposes the first task-oriented text normalization evaluation framework, overcoming the limitations of traditional lemmatization-based metrics by jointly measuring three dimensions: Stemming Effectiveness Score (SES), Model Performance Delta (MPD), and Average Normalized Levenshtein Distance (ANLD). The framework enables integrated analysis of both efficiency and semantic safety—the first of its kind. Empirical evaluation across multiple languages reveals a critical insight: high stemming recall does not necessarily improve downstream performance. For instance, the Bangla stemmer suffers performance degradation due to aggressive over-stemming, whereas the English Snowball stemmer achieves a superior trade-off between effectiveness and semantic fidelity. This work establishes a principled, task-aware methodology for evaluating and comparing stemmers beyond surface-form matching.
📝 Abstract
Text normalization is an essential preprocessing step in many natural language processing (NLP) tasks, and stemming is one such normalization technique that reduces words to their base or root form. However, evaluating stemming methods is challenging because current evaluation approaches are limited and do not capture the potential harm caused by excessive stemming; therefore, it is essential to develop new approaches to evaluate stemming methods. To address this issue, this study propose a novel, task-oriented approach to evaluate stemming methods, which considers three aspects: (1) the utility of stemming using Stemming Effectiveness Score (SES), (2) the impact of stemming on downstream tasks using Model Performance Delta (MPD), and (3) the semantic similarity between stemmed and original words using Average Normalized Levenshtein Distance (ANLD), thus providing a comprehensive evaluation framework. We apply our evaluation framework to compare two stemmers for Bangla (BNLTK) and English (Snowball), and our results reveal a significant issue, prompting us to analyze their performance in detail. While the Bangla stemmer achieves the highest SES (1.67) due to effective word reduction (CR = 1.90), SES alone is insufficient because our proposed safety measure, ANLD, reveals that this high SES is due to harmful over-stemming (ANLD = 0.26), which correlates with the observed decrease in downstream performance.In contrast, the English stemmer achieves a moderate SES (1.31) with a safe meaning distance (ANLD = 0.14), allowing its word reduction to contribute positively to downstream performance; therefore, it is a more reliable stemmer. Our study provides a valuable tool for distinguishing between potential efficiency gains (high SES) and meaning preservation (low ANLD).