UniGen: A Unified Framework for Textual Dataset Generation Using Large Language Models

2024-06-27

arXiv.org

AI Summary

Existing LLM-based text data generation methods suffer from systematic limitations in generalizability, controllability, diversity, and factual fidelity. This paper introduces the first unified LLM framework for general-purpose text dataset generation. It innovatively integrates attribute-guided generation with group-level consistency verification to enhance diversity; combines code-executed mathematical evaluation and retrieval-augmented generation (RAG) to ensure label accuracy and factual consistency; and supports fine-grained, user-specified constraints. The framework unifies GPT-4 and Llama3 backbones with programmable label validation, attribute-conditioned control, and multi-stage collaborative verification. Experiments demonstrate substantial improvements in synthetic data quality—particularly in LLM evaluation benchmark construction and data augmentation tasks—yielding measurable gains in model reasoning performance and agent capabilities, and enabling dynamic, evolution-aware evaluation.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large Language Models (LLMs) such as GPT-4 and Llama3 have significantly impacted various fields by enabling high-quality synthetic data generation and reducing dependence on expensive human-generated datasets. Despite this, challenges remain in the areas of generalization, controllability, diversity, and truthfulness within the existing generative frameworks. To address these challenges, this paper presents UniGen, a comprehensive LLM-powered framework designed to produce diverse, accurate, and highly controllable datasets. UniGen is adaptable, supporting all types of text datasets and enhancing the generative process through innovative mechanisms. To augment data diversity, UniGen incorporates an attribute-guided generation module and a group checking feature. For accuracy, it employs a code-based mathematical assessment for label verification alongside a retrieval-augmented generation technique for factual validation. The framework also allows for user-specified constraints, enabling customization of the data generation process to suit particular requirements. Extensive experiments demonstrate the superior quality of data generated by UniGen, and each module within UniGen plays a critical role in this enhancement. Additionally, UniGen is applied in two practical scenarios: benchmarking LLMs and data augmentation. The results indicate that UniGen effectively supports dynamic and evolving benchmarking, and that data augmentation improves LLM capabilities in various domains, including agent-oriented abilities and reasoning skills.

Problem

• Addressing generalization and controllability in synthetic data generation
• Enhancing diversity and truthfulness in LLM-generated datasets
• Providing customizable data generation for specific user requirements

Innovation

• Attribute-guided generation for diversity
• Code-based math assessment for accuracy
• User-specified constraints for customization