Large corpora and large language models: a replicable method for automating grammatical annotation

2024-11-18

arXiv.org

AI Summary

To address the low efficiency and high labor cost of syntactic annotation in large-scale corpora, this paper proposes a reproducible supervised large language model (LLM)-assisted annotation framework. We pioneer the application of Claude 3.5 Sonnet to formal variation annotation of English evaluative verb constructions, integrating prompt engineering, few-shot supervised fine-tuning, and human verification. The end-to-end, reusable pipeline is implemented on the Davies NOW and EnTenTen21 corpora. Our approach achieves high annotation accuracy—exceeding 90% on held-out test sets—while preserving linguistic interpretability and requiring only minimal annotated data for high-throughput construction identification. The resulting methodology provides an efficient, reliable, and scalable technical foundation for grammatical variation analysis and diachronic linguistic research.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Much linguistic research relies on annotated datasets of features extracted from text corpora, but the rapid quantitative growth of these corpora has created practical difficulties for linguists to manually annotate large data samples. In this paper, we present a replicable, supervised method that leverages large language models for assisting the linguist in grammatical annotation through prompt engineering, training, and evaluation. We introduce a methodological pipeline applied to the case study of formal variation in the English evaluative verb construction 'consider X (as) (to be) Y', based on the large language model Claude 3.5 Sonnet and corpus data from Davies' NOW and EnTenTen21 (SketchEngine). Overall, we reach a model accuracy of over 90% on our held-out test samples with only a small amount of training data, validating the method for the annotation of very large quantities of tokens of the construction in the future. We discuss the generalisability of our results for a wider range of case studies of grammatical constructions and grammatical variation and change, underlining the value of AI copilots as tools for future linguistic research, notwithstanding some important caveats.

Problem

• Automating grammatical annotation for large text corpora
• Reducing manual effort in linguistic feature extraction
• Validating AI-assisted annotation accuracy for grammatical constructions

Innovation

• Leveraging LLMs for grammatical annotation
• Using Claude 3.5 Sonnet model
• Achieving 90% accuracy with minimal training