A large-scale, unsupervised pipeline for automatic corpus annotation using LLMs: variation and change in the English consider construction

2025-10-14

AI Summary

To address the low efficiency and high cost of manual grammatical annotation in corpus linguistics, this paper proposes an LLM-driven unsupervised grammatical annotation framework requiring no labeled data. Methodologically, we design a four-stage pipeline comprising prompt engineering, pre-evaluation filtering, batched API invocation, and pre-/post-annotation validation, leveraging GPT-5 as the foundational model to achieve end-to-end automated annotation on the Corpus of Historical American English (COHA). Our key contribution is the first systematic integration of large language models into large-scale diachronic grammatical analysis, balancing annotation accuracy with computational scalability. Experimental results demonstrate that the framework annotates 143,000 sentences within 60 hours at >98% accuracy, drastically reducing human intervention and temporal overhead. This work establishes a reproducible, robust paradigm for large-scale, evolution-oriented linguistic research grounded in historical corpora.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

As natural language corpora expand at an unprecedented rate, manual annotation remains a significant methodological bottleneck in corpus linguistic work. We address this challenge by presenting a scalable, unsupervised pipeline for automating grammatical annotation in voluminous corpora using large language models (LLMs). Unlike previous supervised and iterative approaches, our method employs a four-phase workflow: prompt engineering, pre-hoc evaluation, automated batch processing, and post-hoc validation. We demonstrate the pipeline's accessibility and effectiveness through a diachronic case study of variation in the English consider construction. Using GPT-5 through the OpenAI API, we annotate 143,933 sentences from the Corpus of Historical American English (COHA) in under 60 hours, achieving 98%+ accuracy on two sophisticated annotation procedures. Our results suggest that LLMs can perform a range of data preparation tasks at scale with minimal human intervention, opening new possibilities for corpus-based research, though implementation requires attention to costs, licensing, and other ethical considerations.

Problem

• Automating corpus annotation using LLMs to overcome manual limitations
• Developing unsupervised pipeline for grammatical annotation in large corpora
• Addressing variation in English consider construction through automated analysis

Innovation

• Unsupervised LLM pipeline automates grammatical annotation
• Four-phase workflow ensures scalable corpus processing
• GPT-5 achieves high accuracy in batch annotation