Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

2024-03-28

arXiv.org

AI Summary

Large-scale text cleaning, filtering, and formatting for large language model (LLM) data engineering face high technical barriers and suffer from fragmented, non-integrated tooling. Method: This paper introduces the first low-code, block-based open-source ETL framework specifically designed for LLM data engineering. It features a modular, configuration-driven pipeline architecture with pluggable processor interfaces, enabling rapid integration of custom data processing logic; supports both CLI and programmatic API invocation to balance usability and flexibility. Contribution/Results: Experiments demonstrate efficient automated preprocessing of TB-scale corpora, significantly reducing LLM data preparation overhead. The framework is fully open-sourced, accompanied by tutorial videos and comprehensive documentation, thereby advancing standardization and community-driven development in LLM data engineering.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of custom processors with block-based interface in Dataverse allows users to readily and efficiently use Dataverse to build their own ETL pipeline. We hope that Dataverse will serve as a vital tool for LLM development and open source the entire library to welcome community contribution. Additionally, we provide a concise, two-minute video demonstration of our system, illustrating its capabilities and implementation.

Problem

• Addresses challenges in large-scale data processing for LLMs.
• Proposes a user-friendly, open-source ETL pipeline for LLMs.
• Enables easy integration of custom processors for efficient ETL workflows.

Innovation

• Open-source ETL pipeline for LLMs
• Block-based interface for custom processors
• User-friendly design for efficient data processing