GliLem: Leveraging GliNER for Contextualized Lemmatization in Estonian

2024-12-29

AI Summary

To address the low lemmatization accuracy of Estonian text—which hinders information retrieval performance—this paper proposes a contextualized hybrid tokenization method. Building upon the rule-based morphological analyzer Vabamorf, we introduce the open-vocabulary NER model GliNER as an external disambiguation module for context-aware lemmatization via natural language label matching. Our key contributions are: (1) the first application of GliNER to lemmatization disambiguation; (2) a novel contextualized lemmatization paradigm; and (3) the construction of the first Estonian information retrieval benchmark dataset. Experiments show that our method improves lemmatization accuracy by 10% over the baseline Vabamorf, achieves significantly higher recall at high k-values in retrieval tasks compared to standard baselines, and consistently outperforms traditional stemming approaches.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

We present GliLem -- a novel hybrid lemmatization system for Estonian that enhances the highly accurate rule-based morphological analyzer Vabamorf with an external disambiguation module based on GliNER -- an open vocabulary NER model that is able to match text spans with text labels in natural language. We leverage the flexibility of a pre-trained GliNER model to improve the lemmatization accuracy of Vabamorf by 10% compared to its original disambiguation module and achieve an improvement over the token classification-based baseline. To measure the impact of improvements in lemmatization accuracy on the information retrieval downstream task, we first created an information retrieval dataset for Estonian by automatically translating the DBpedia-Entity dataset from English. We benchmark several token normalization approaches, including lemmatization, on the created dataset using the BM25 algorithm. We observe a substantial improvement in IR metrics when using lemmatization over simplistic stemming. The benefits of improving lemma disambiguation accuracy manifest in small but consistent improvement in the IR recall measure, especially in the setting of high k.

Problem

• Estonian language
• word stem recognition
• information retrieval

Innovation

• GliLem
• root-word recognition
• tokenization method