Towards Building Private LLMs: Exploring Multi-Node Expert Parallelism on Apple Silicon for Mixture-of-Experts Large Language Model

2025-06-30

AI Summary

Deploying private large language models (LLMs) in personal and small-scale settings faces prohibitive costs and scalability bottlenecks. Method: This paper proposes a multi-node expert-parallel inference framework tailored for Apple Silicon, deploying the MoE-based DBRX model on an M2 Ultra Mac Studio cluster. It identifies network latency as a critical bottleneck in distributed MoE inference, designs memory-management optimizations to eliminate redundant overheads in Apple’s software stack, and develops a lightweight performance modeling methodology for cross-configuration inference latency and throughput prediction. Contribution/Results: Experimental evaluation demonstrates that the proposed solution reduces total cost by 15% compared to an NVIDIA H100 supercomputing system while significantly improving end-to-end inference speed. It establishes the first high-performance, low-cost, and predictable private LLM deployment paradigm native to the Apple ecosystem.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Distributed Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI) with significant advancements such as OpenAI's ChatGPT, Meta's Llama, and Databricks' DBRX. This paper addresses the cost and scalability challenges encountered when constructing private LLM systems for personal or small group services, as aimed by Apple Intelligence. A Mac Studio cluster with Apple's M2 Ultra chips is established as a cost-efficient solution to host and accelerate the pretrained DBRX model with the Mixture-of-Experts (MoE) architecture. Our performance analysis reveal that parallel execution of the model's experts across two to four machine nodes significantly reduces inference time. We find that computation time for the experts is comparable to the communication time for exchanging their outputs, emphasizing the importance of network latency over bandwidth. We also observe significant management overhead due to Apple software stack's memory management logic. Based on these findings, we develop optimization schemes to eliminate the memory management overhead. As a result, the Mac Studio cluster is 1.15 times more cost-efficient than the state-of-the-art AI supercomputer with NVIDIA H100 GPUs. In addition, we construct a performance model to estimate system performance under varying configurations, and the model provides valuable insights for designing private LLM systems.

Problem

• Addressing cost and scalability in private LLM systems
• Optimizing multi-node expert parallelism on Apple Silicon
• Reducing inference time and management overhead in MoE models

Innovation

• Multi-node expert parallelism on Apple Silicon
• Optimized memory management for efficiency
• Performance modeling for private LLM systems