TGEA: An Error-Annotated Dataset and Benchmark Tasks for TextGeneration from Pretrained Language Models

2025-03-06

Annual Meeting of the Association for Computational Linguistics

AI Summary

This work addresses the challenge of diagnosing errors in text generated by pretrained language models (PLMs). We introduce TGEA, the first fine-grained error annotation dataset specifically designed for large language model (LLM) outputs, comprising 47K generated sentences and 12K human-annotated error instances spanning 24 linguistically and knowledge-based error categories. TGEA pioneers a multidimensional annotation schema—including error span, associated span, minimal correction, error type, and root-cause analysis—grounded in GPT-2–based controllable generation, crowdsourced annotation, and a linguistics- and commonsense-informed taxonomy. Leveraging this dataset, we establish a comprehensive PLM generation diagnostics benchmark, formalizing novel tasks: error detection, fine-grained classification, association identification, and attribution generation. TGEA substantially enhances the interpretability and diagnosability of PLM-generated errors, providing both foundational resources and standardized evaluation protocols for automated error detection and correction research.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

In order to deeply understand the capability of pretrained language models in text generation and conduct a diagnostic evaluation, we propose TGEA, an error-annotated dataset with multiple benchmark tasks for text generation from pretrained language models (PLMs). We use carefully selected prompt words to guide GPT-2 to generate candidate sentences, from which we select 47K for error annotation. Crowdsourced workers manually check each of these sentences and detect 12k erroneous sentences. We create an error taxonomy to cover 24 types of errors occurring in these erroneous sentences according to the nature of errors with respect to linguistics and knowledge (e.g., common sense). For each erroneous span in PLM-generated sentences, we also detect another span that is closely associated with it. Each error is hence manually labeled with comprehensive annotations, including the span of the error, the associated span, minimal correction to the error, the type of the error, and rationale behind the error. Apart from the fully annotated dataset, we also present a detailed description of the data collection procedure, statistics and analysis of the dataset. This is the first dataset with comprehensive annotations for PLM-generated texts, which facilitates the diagnostic evaluation of PLM-based text generation. Furthermore, we use TGEA as a benchmark dataset and propose a series of automatic diagnosis tasks, including error detection, error type classification, associated span detection, error rationale generation, to further promote future study on the automatic error detection and correction on texts generated by pretrained language models.

Problem

• Develops TGEA dataset for evaluating pretrained language models' text generation.
• Annotates 47K sentences for errors, identifying 12K with 24 error types.
• Proposes benchmark tasks for automatic error detection and correction in PLM texts.

Innovation

• Error-annotated dataset for PLM text generation
• Crowdsourced manual error detection and taxonomy
• Benchmark tasks for automatic error diagnosis