ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

2026-03-08

AI Summary

This work addresses the limited scalability of existing large language model (LLM) inference frameworks on multi-core CPUs, which is primarily hindered by the high overhead of cross-NUMA-node memory accesses. To overcome this challenge, we propose the first lightweight LLM inference framework explicitly optimized for NUMA-aware CPU architectures. By integrating NUMA-aware thread scheduling, efficient memory management, and fine-grained tensor parallelism, our approach substantially reduces inter-node communication costs. The design maintains full compatibility with arbitrary CPU devices while achieving up to a 46% improvement in inference throughput, significantly surpassing the performance ceiling of current mainstream frameworks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.

Problem

• LLM inference
• many-core CPUs
• NUMA
• memory access overhead
• scalability

Innovation

• many-core CPUs
• NUMA-aware
• tensor parallelism
• LLM inference
• memory management