LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMs

2024-08-24

arXiv.org

AI Summary

To address cloud dependency, privacy leakage, and network outage risks, this paper proposes a knowledge transfer framework for localized LLM deployment. Our method introduces a novel multi-round self-enhancing fine-tuning paradigm: high-quality synthetic data—generated by service-based large language models (e.g., GPT-4, Claude)—drives iterative supervised fine-tuning and knowledge distillation of lightweight models (e.g., Phi-3, Qwen2, <3B parameters). An LLMOps-powered automated pipeline orchestrates and evaluates the entire process. Experiments demonstrate that the fine-tuned small models match or surpass the original service models across multiple downstream tasks, while reducing inference cost by over 90%. Crucially, they enable fully offline, privacy-preserving, and network-outage-resilient deployment. This significantly enhances the robustness and controllability of AI services without compromising performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The widespread adoption of cloud-based proprietary large language models (LLMs) has introduced significant challenges, including operational dependencies, privacy concerns, and the necessity of continuous internet connectivity. In this work, we introduce an LLMOps pipeline,"LlamaDuo", for the seamless migration of knowledge and abilities from service-oriented LLMs to smaller, locally manageable models. This pipeline is crucial for ensuring service continuity in the presence of operational failures, strict privacy policies, or offline requirements. Our LlamaDuo involves fine-tuning a small language model against the service LLM using a synthetic dataset generated by the latter. If the performance of the fine-tuned model falls short of expectations, it is automatically improved through additional fine-tuning using extra similar data generated by the service LLM. This multi-turn process guarantees that the smaller model can eventually match or even surpass the service LLM's capabilities in specific downstream tasks, offering a practical and scalable solution for managing AI deployments in constrained environments. Extensive experiments with leading-edge LLMs are conducted to demonstrate the effectiveness, adaptability, and affordability of LlamaDuo across various downstream tasks. Our pipeline implementation is available at https://github.com/deep-diver/llamaduo.

Problem

• Migrate knowledge from cloud LLMs to local small-scale models
• Address privacy and offline operation challenges in LLM deployment
• Automatically improve small models to match service LLM performance

Innovation

• LLMOps pipeline for seamless migration
• Fine-tuning small models with synthetic data
• Multi-turn process ensures performance matching