DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models

2026-04-19

AI Summary

This work addresses the bottleneck of low parallel inference efficiency that hinders the scalability of existing decomposed large language models. To overcome this limitation, we propose DeInfer, the first high-performance parallel inference system tailored for decomposed large language models. DeInfer systematically enhances inference throughput and scalability through a co-design of multi-level parallelism strategies, communication optimizations, and integration with established acceleration techniques. Experimental results demonstrate that DeInfer consistently outperforms current baselines across diverse configurations, effectively enabling efficient and scalable parallel inference for large-scale decomposed models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Existing works on large language model (LLM) decomposition mainly focus on improving performance on downstream tasks, but they ignore the poor parallel inference performance when trying to scale up the model size. To mitigate this important performance issue, this paper introduces DeInfer, a high-performance inference system dedicated to parallel inference of decomposed LLMs. It consists of multiple optimizations to maximize performance and be compatible with state-of-the-art optimization techniques. Extensive experiments are carried out to evaluate DeInfer's performance, where the results demonstrate its superiority, suggesting it can greatly facilitate the parallel inference of decomposed LLMs.

Problem

• large language models
• model decomposition
• parallel inference
• inference performance
• scalability

Innovation

• parallel inference
• model decomposition
• large language models
• high-performance system
• efficient inference