PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detection

2024-12-16

arXiv.org

AI Summary

To address the sensitivity of in-context learning (ICL) to example selection, the scarcity of human annotations, and the poorly understood generalization mechanisms in low-resource named entity detection (NED), this paper proposes PICLe: a framework that requires no manual labeling. PICLe leverages large language models (LLMs) for zero-shot generation of noisy pseudo-labeled in-context examples; integrates semantic clustering-based sampling with multi-path parallel inference; and introduces a self-verification mechanism to dynamically filter high-confidence predictions and fuse decisions. A key finding is that partially correct pseudo-labeled examples achieve ICL performance comparable to fully correct ones. Evaluated on five biomedical NER datasets, PICLe significantly outperforms standard ICL under zero-shot and few-shot settings—especially when only a small number of gold examples are available—demonstrating robust, scalable contextual generalization for low-resource NED without human annotation.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Neuro-Symbolic Learning
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In-context learning (ICL) enables Large Language Models (LLMs) to perform tasks using few demonstrations, facilitating task adaptation when labeled examples are hard to obtain. However, ICL is sensitive to the choice of demonstrations, and it remains unclear which demonstration attributes enable in-context generalization. In this work, we conduct a perturbation study of in-context demonstrations for low-resource Named Entity Detection (NED). Our surprising finding is that in-context demonstrations with partially correct annotated entity mentions can be as effective for task transfer as fully correct demonstrations. Based off our findings, we propose Pseudo-annotated In-Context Learning (PICLe), a framework for in-context learning with noisy, pseudo-annotated demonstrations. PICLe leverages LLMs to annotate many demonstrations in a zero-shot first pass. We then cluster these synthetic demonstrations, sample specific sets of in-context demonstrations from each cluster, and predict entity mentions using each set independently. Finally, we use self-verification to select the final set of entity mentions. We evaluate PICLe on five biomedical NED datasets and show that, with zero human annotation, PICLe outperforms ICL in low-resource settings where limited gold examples can be used as in-context demonstrations.

Problem

• Investigates effectiveness of partially correct annotations in NED
• Proposes PICLe for ICL with noisy pseudo-annotated demonstrations
• Enhances low-resource NED using zero-shot LLM annotations

Innovation

• Uses pseudo-annotated demonstrations for ICL
• Clusters synthetic demonstrations for sampling
• Applies self-verification for final predictions