Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition

2026-06-20

AI Summary

This study addresses the persistent performance gap between large language models (LLMs) and supervised fine-tuned models in named entity recognition (NER), noting that few-shot in-context learning fails to harness the full potential of abundant examples. The authors systematically investigate many-shot in-context learning for NER and demonstrate, for the first time, that providing hundreds of contextual examples enables LLMs to match or even surpass the performance of fine-tuned BERT models. Building on this finding, they propose a novel high-quality data annotation paradigm tailored for low-resource settings: by augmenting only around one hundred manually labeled samples to generate enriched training corpora, the F1 score of fine-tuned BERT improves by approximately 10%.

Abstract

In-context learning (ICL) with large language models (LLMs) has emerged as a powerful alternative to fine-tuning for Named Entity Recognition (NER), achieving strong performance with minimal annotation and no additional training. However, prior work has shown that despite their adaptability, LLMs still lag behind fully supervised models such as fine-tuned BERT in structured tasks like NER. While existing studies on ICL for NER have mainly explored few-shot settings, the potential of scaling to hundreds of demonstrations has not been thoroughly investigated. To address this gap, we conduct a comprehensive investigation of many-shot ICL for NER and further explore its effectiveness in annotating and refining data for low-resource NER tasks. Specifically, we evaluate various LLMs across multiple domains using hundreds of ICL examples and then assess the feasibility of using many-shot ICL as a data annotation framework. Our experiments demonstrate that: (1) scaling to hundreds of in-context examples enables LLMs to match or even surpass the performance of fully supervised BERT models; and (2) using about one hundred human-labeled examples as demonstrations, many-shot in-context annotation can generate high-quality labeled data, leading to approximately 10% absolute F1 improvement over existing state-of-the-art approaches when used to fine-tune BERT on low-resource NER.

Problem

• Named Entity Recognition
• In-Context Learning
• Low-Resource Annotation
• Many-Shot Learning
• Large Language Models

Innovation

• many-shot in-context learning
• named entity recognition
• low-resource annotation
• large language models
• data augmentation