Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation

2024-11-27

arXiv.org

AI Summary

Traditional word sense disambiguation (WSD) methods suffer from limited performance in digital communications due to lexical ambiguity and severe scarcity of annotated training data. Method: This paper proposes a human-in-the-loop prompt engineering framework that innovatively integrates human-guided prompt enhancement, fine-grained part-of-speech (POS) tagging, synonym-based semantic expansion, sense-dimensional filtering, and few-shot chain-of-thought prompting. Contribution/Results: The framework significantly improves large language models’ (LLMs) fine-grained semantic discrimination capability for ambiguous words. Evaluated on the FEWS benchmark, our approach achieves substantial accuracy gains over strong baselines, markedly enhancing robustness in disambiguating polysemous terms within social media and digital texts. It thereby provides more reliable semantic foundations for downstream applications—including machine translation, information retrieval, and question answering—without requiring extensive labeled corpora.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Ambiguous words are often found in modern digital communications. Lexical ambiguity challenges traditional Word Sense Disambiguation (WSD) methods, due to limited data. Consequently, the efficiency of translation, information retrieval, and question-answering systems is hindered by these limitations. This study investigates the use of Large Language Models (LLMs) to improve WSD using a novel approach combining a systematic prompt augmentation mechanism with a knowledge base (KB) consisting of different sense interpretations. The proposed method incorporates a human-in-loop approach for prompt augmentation where prompt is supported by Part-of-Speech (POS) tagging, synonyms of ambiguous words, aspect-based sense filtering and few-shot prompting to guide the LLM. By utilizing a few-shot Chain of Thought (COT) prompting-based approach, this work demonstrates a substantial improvement in performance. The evaluation was conducted using FEWS test data and sense tags. This research advances accurate word interpretation in social media and digital communication.

Problem

• Evaluating LLMs for Word Sense Disambiguation (WSD) in digital communications
• Improving WSD using prompt augmentation and knowledge base integration
• Enhancing translation and information retrieval via LLM-based ambiguity resolution

Innovation

• Combines prompt augmentation with knowledge base
• Uses human-in-loop approach for prompt refinement
• Applies few-shot Chain of Thought prompting