HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing

2026-04-20

AI Summary

This work addresses the significant memory and bandwidth pressure imposed by key-value (KV) caching in long-context reasoning with large language models, as well as the underutilization of hardware resources due to reliance on a single processor for attention computation. To overcome these challenges, the authors propose HybridGen, a novel framework that synergistically integrates CPU-GPU collaborative computation, CXL-enabled memory expansion, logit-level attention parallelism, a feedback-driven scheduler, and semantic-aware KV cache mapping. This design effectively mitigates multidimensional dependencies, load imbalance, and NUMA-induced latency. Extensive experiments across three GPU platforms and eleven models demonstrate that HybridGen achieves 1.41–3.2× higher average inference throughput compared to six state-of-the-art KV cache management approaches while preserving high model accuracy.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Generation
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

As modern LLMs support thousands to millions of tokens, KV caches grow to hundreds of gigabytes, stressing memory capacity and bandwidth. Existing solutions, such as KV cache pruning and offloading, alleviate these but underutilize hardware by relying solely on either GPU or CPU for attention computing, and considering yet limited CPU local memory for KV cache storage. We propose HybridGen, an efficient hybrid attention framework for long-context LLM inference. HybridGen enables CPU-GPU collaborative attention on systems with expanded tiered memory (e.g., CXL memory), addressing three key challenges: (1) multi-dimensional attention dependencies, (2) intensifying CPU-GPU load imbalance with longer sequences, and (3) NUMA penalty of tiered memories. HybridGen tackles these by introducing attention logit parallelism, a feedback-driven scheduler, and semantic-aware KV cache mapping. Experiments with three LLM models with eleven different sizes on three GPU platforms with a CXL-expanded memory show that HybridGen outperforms six state-of-the-art KV cache management methods by 1.41x--3.2x on average while maintaining superior accuracy.

Problem

• LLM inference
• KV cache
• CPU-GPU hybrid computing
• long-context
• memory bottleneck

Innovation

• Hybrid Computing
• KV Cache Management
• Long-Context LLM Inference
• CPU-GPU Collaboration
• Tiered Memory