HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloads

2026-08-19

AI Summary

为解决混合大语言模型在异构芯片系统上高效运行的问题，HYDRA框架通过探索芯片组合、放置、带宽分配及动态调度等方法优化硬件加速。

Abstract

Hybrid Transformer-Mamba large language models (LLMs) enhance long-context efficiency, but their heterogeneous computation and communication patterns complicate efficient hardware acceleration. Chiplet-based architectures offer a scalable solution by integrating specialized compute and memory units. However, the design space spanning static architectural configurations and dynamic runtime policies is prohibitively large to explore exhaustively. To address this challenge, we present HYDRA, a comprehensive design space exploration framework for hybrid LLM serving on heterogeneous chiplet systems. HYDRA jointly explores chiplet composition, placement, inter-chiplet bandwidth provisioning, dynamic batching, and runtime scheduling. It integrates communication-aware placement, dynamic batching, elastic task scheduling, and a fast Markov-based performance estimator that captures multi-tenant runtime dynamics for efficient and accurate exploration. Across all workloads, HYDRA delivers 1.55x the throughput and 43.7 percent lower time-to-first-token on average, with throughput gains reaching up to 2.3x compared to state-of-the-art baselines. These results highlight that co-designing architecture and runtime policies is critical for efficient large-scale LLM serving on heterogeneous chiplet systems.

Problem

• Heterogeneous Chiplet
• Large Language Models
• Design Space Exploration
• Dynamic Workloads
• Hardware Acceleration

Innovation

• Heterogeneous Chiplet
• Design Space Exploration
• Dynamic Batching
• Communication-aware Placement
• Markov-based Performance Estimator