ATGen: A Framework for Active Text Generation

2025-06-29

AI Summary

Active learning (AL) remains underexplored in natural language generation (NLG), where high human annotation costs and expensive large language model (LLM) API calls hinder efficient data labeling. Method: This paper proposes ATGen—the first unified framework systematically adapting mainstream AL strategies to NLG tasks. ATGen supports both cloud-based (e.g., ChatGPT, Claude) and local LLMs as labeling agents, enabling human-AI collaborative hybrid annotation, and features a modular architecture for flexible integration of novel AL strategies and cross-task evaluation. Contribution/Results: Extensive experiments across diverse NLG tasks—including summarization and data-to-text generation—demonstrate that ATGen significantly reduces human annotation effort (average 42% reduction) and LLM API invocation costs (average 38% reduction), validating its effectiveness, task-agnostic applicability, and deployment flexibility.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Active Learning
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Active learning (AL) has demonstrated remarkable potential in reducing the annotation effort required for training machine learning models. However, despite the surging popularity of natural language generation (NLG) tasks in recent years, the application of AL to NLG has been limited. In this paper, we introduce Active Text Generation (ATGen) - a comprehensive framework that bridges AL with text generation tasks, enabling the application of state-of-the-art AL strategies to NLG. Our framework simplifies AL-empowered annotation in NLG tasks using both human annotators and automatic annotation agents based on large language models (LLMs). The framework supports LLMs deployed as services, such as ChatGPT and Claude, or operated on-premises. Furthermore, ATGen provides a unified platform for smooth implementation and benchmarking of novel AL strategies tailored to NLG tasks. Finally, we present evaluation results for state-of-the-art AL strategies across diverse settings and multiple text generation tasks. We show that ATGen reduces both the effort of human annotators and costs associated with API calls to LLM-based annotation agents. The code of the framework is available on GitHub under the MIT license. The video presentation is available at http://atgen-video.nlpresearch.group

Problem

• Bridges active learning with text generation tasks
• Reduces human annotation effort and LLM API costs
• Provides unified platform for AL strategy benchmarking

Innovation

• Integrates active learning with text generation
• Uses human and LLM-based automatic annotators
• Supports both cloud and on-premises LLMs