CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory

2025-03-04

AI Summary

Deploying Collaborative Experts (CoE) models in resource-constrained settings faces critical bottlenecks—excessive GPU memory consumption and high overhead from frequent expert switching. Method: This work introduces, for the first time, an expert dependency modeling framework that constructs an expert dependency graph to characterize inter-task expert invocation patterns. Based on this graph, we design a dependency-aware request scheduler and a dynamic expert loading/unloading mechanism. Further, leveraging offline performance profiling across heterogeneous CPU/GPU hardware, we formulate a memory-aware automatic resource allocation optimization. Contribution/Results: Evaluated on real-world smart manufacturing PCB inspection workloads, our system achieves 4.5×–12× higher throughput than state-of-the-art CoE deployment methods, while significantly reducing expert switching frequency and peak GPU memory usage. This establishes a new paradigm for high-accuracy, resource-efficient multi-expert collaborative inference.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models like GPT-4 are resource-intensive, but recent advancements suggest that smaller, specialized experts can outperform the monolithic models on specific tasks. The Collaboration-of-Experts (CoE) approach integrates multiple expert models, improving the accuracy of generated results and offering great potential for precision-critical applications, such as automatic circuit board quality inspection. However, deploying CoE serving systems presents challenges to memory capacity due to the large number of experts required, which can lead to significant performance overhead from frequent expert switching across different memory and storage tiers. We propose CoServe, an efficient CoE model serving system on heterogeneous CPU and GPU with limited memory. CoServe reduces unnecessary expert switching by leveraging expert dependency, a key property of CoE inference. CoServe introduces a dependency-aware request scheduler and dependency-aware expert management for efficient inference. It also introduces an offline profiler to automatically find optimal resource allocation on various processors and devices. In real-world intelligent manufacturing workloads, CoServe achieves 4.5$ imes$ to 12$ imes$ higher throughput compared to state-of-the-art systems.

Problem

• Optimizes memory usage in Collaboration-of-Experts model inference.
• Reduces performance overhead from frequent expert switching.
• Enhances throughput in precision-critical applications like manufacturing.

Innovation

• Dependency-aware request scheduler for CoE
• Dependency-aware expert management system
• Offline profiler for optimal resource allocation