Towards Scalable Customization and Deployment of Multi-Agent Systems for Enterprise Applications

2026-06-16

AI Summary

Enterprise-scale multi-agent systems often face challenges including domain adaptation difficulties, high inference latency, and substantial deployment costs. This work proposes a unified framework that enables efficient customization and deployment through a two-stage optimization strategy. First, domain adaptation is achieved by integrating continual pre-training, supervised fine-tuning, and preference optimization. Subsequently, low-overhead, high-efficiency inference is realized via speculative decoding, FP8 quantization, and targeted calibration. Notably, this approach co-designs domain-specific customization of compact models with inference optimization for multi-agent workflows—a first in the field. The method maintains task performance while significantly improving throughput by up to 4.48× and enhancing robustness in long-tail scenarios.

Abstract

Large language model (LLM)-based multi-agent systems demonstrate strong performance on complex reasoning and task execution, enabling broad enterprise applications. However, production deployment remains challenging due to domain-specific customization requirements and high latency and inference costs in agentic workflows. We propose a unified framework for customization and efficient deployment of multi-agent systems in real-world settings. The first stage, Agentic Model Customization, combines continual pretraining, supervised fine-tuning, and preference optimization to adapt a compact model to specialized domains while retaining strong agentic capabilities. The second stage, Inference Optimization, integrates speculative decoding and FP8 quantization with targeted calibration to enable cost-efficient serving with minimal quality loss. Across enterprise workloads, our framework enables rapid domain adaptation and achieves a 4.48x speedup in throughput while maintaining performance and improving robustness on long-tail scenarios.

Problem

• multi-agent systems
• enterprise applications
• domain customization
• inference cost
• latency

Innovation

• multi-agent systems
• model customization
• speculative decoding
• FP8 quantization
• enterprise deployment