🤖 AI Summary
This work addresses the challenge that large language models (LLMs) struggle to adhere to domain-specific gold-standard annotation guidelines in zero-shot settings. To mitigate this limitation, the authors propose a mediation framework that iteratively reuses and refines annotation guidelines, introducing guideline evolution as a novel alignment mechanism to enhance annotation consistency and accuracy under low-supervision conditions. The approach integrates reasoning-optimized variants from three major LLM families—GPT, Gemini, and DeepSeek—and employs iterative guideline consolidation and fine-tuning. Evaluated on biomedical named entity recognition benchmarks including NCBI Disease, BC5CDR, and BioRED, the method demonstrates significant improvements in the models’ compliance with expert annotation standards.
📝 Abstract
While Large Language Models (LLMs) demonstrate remarkable performance on zero-shot annotation tasks, they often struggle with the specialized conventions of gold-standard benchmarks. We propose the systematic reuse and refinement of annotation guidelines as an alignment mechanism, introducing an iterative moderation framework that simulates the early phases of annotation projects. We evaluate three hypotheses: (1) the efficacy of guideline integration, (2) the advantage of reasoning optimized models, and (3) the viability of moderation under minimal supervision. Testing across biomedical NER tasks (NCBI Disease, BC5CDR, BioRED) with three LLM families (GPT, Gemini, DeepSeek), our results empirically confirm all three hypotheses. While the iterative moderation framework shows good potential in effectively refining guidelines, our analysis also reveals substantial room for improvement.