Institution profile

SK hynix Inc.

Industry researchasia · kr
Official website
Research library28linked papers
Opportunities0open roles
Selected work

Representative Papers

NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement

Jul 29, 2026

This work addresses the throughput and latency bottlenecks of conventional GPU architectures when handling mixed-length LLM inference requests with highly variable context lengths. The authors propose the first heterogeneous system integrating real near-memory computing (PNM) with GPUs via CXL interconnects, co-designing sparse attention execution on PNM, decoding kernels on GPUs, and a host-based scheduling runtime. They introduce a request-length-aware placement strategy and a dynamic migration mechanism that avoids recomputation, enabling efficient cross-device memory management with RPC/RDMA support. Under mixed-length workloads, the system achieves up to 5.5× higher decoding throughput and reduces P99 latency by up to 15× compared to GPU-only baselines.

0 citationsRead paper

HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory

Jul 20, 2026

This work addresses the scalability limitations of KV cache reuse in multi-turn large language model (LLM) serving, which is constrained by the capacity and cost of GPU HBM and host DRAM, hindering terabyte-scale shared context support. To overcome this, the authors propose HyMCache, a novel framework that introduces CXL-based heterogeneous memory—combining a small amount of device DRAM with high-capacity SSDs—into multi-turn LLM inference. HyMCache enables efficient KV cache reuse through request-level prefix prefetching, opportunistic write buffering, and a reengineered DRAM management policy. Under identical DRAM budgets, HyMCache achieves 3.0× higher throughput per node than LMCache. Compared to Mooncake—a 1TB distributed DRAM solution—it reduces DRAM usage by 16× while incurring only ~30% performance degradation, effectively balancing high capacity, low cost, and low latency.

0 citationsRead paper

StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

Jul 09, 2026

This work addresses the throughput bottleneck in large model inference caused by dequantization on GPUs, which incurs high instruction overhead, surges in on-chip traffic, and pipeline stalls. The authors propose the first near-memory dequantization approach by integrating a lightweight dequantization module into the HBM base die. This design leverages sideband tags attached to memory read requests to dynamically select operating modes, enabling “load-and-dequantize” while preserving standard load semantics. Coupled with a customized HBM architecture, on-die dequantization blocks (DQBs), and mixed-precision GEMM acceleration, the method achieves up to a 7.08× speedup in GEMM computation, 90.23% energy reduction, 54.68% lower end-to-end LLM inference latency, and a 2.20× improvement in decoding throughput.

0 citationsRead paper
Recent publications

Latest Papers

NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement

Jul 29, 2026

This work addresses the throughput and latency bottlenecks of conventional GPU architectures when handling mixed-length LLM inference requests with highly variable context lengths. The authors propose the first heterogeneous system integrating real near-memory computing (PNM) with GPUs via CXL interconnects, co-designing sparse attention execution on PNM, decoding kernels on GPUs, and a host-based scheduling runtime. They introduce a request-length-aware placement strategy and a dynamic migration mechanism that avoids recomputation, enabling efficient cross-device memory management with RPC/RDMA support. Under mixed-length workloads, the system achieves up to 5.5× higher decoding throughput and reduces P99 latency by up to 15× compared to GPU-only baselines.

0 citationsRead paper

HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory

Jul 20, 2026

This work addresses the scalability limitations of KV cache reuse in multi-turn large language model (LLM) serving, which is constrained by the capacity and cost of GPU HBM and host DRAM, hindering terabyte-scale shared context support. To overcome this, the authors propose HyMCache, a novel framework that introduces CXL-based heterogeneous memory—combining a small amount of device DRAM with high-capacity SSDs—into multi-turn LLM inference. HyMCache enables efficient KV cache reuse through request-level prefix prefetching, opportunistic write buffering, and a reengineered DRAM management policy. Under identical DRAM budgets, HyMCache achieves 3.0× higher throughput per node than LMCache. Compared to Mooncake—a 1TB distributed DRAM solution—it reduces DRAM usage by 16× while incurring only ~30% performance degradation, effectively balancing high capacity, low cost, and low latency.

0 citationsRead paper

StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

Jul 09, 2026

This work addresses the throughput bottleneck in large model inference caused by dequantization on GPUs, which incurs high instruction overhead, surges in on-chip traffic, and pipeline stalls. The authors propose the first near-memory dequantization approach by integrating a lightweight dequantization module into the HBM base die. This design leverages sideband tags attached to memory read requests to dynamically select operating modes, enabling “load-and-dequantize” while preserving standard load semantics. Coupled with a customized HBM architecture, on-die dequantization blocks (DQBs), and mixed-precision GEMM acceleration, the method achieves up to a 7.08× speedup in GEMM computation, 90.23% energy reduction, 54.68% lower end-to-end LLM inference latency, and a 2.20× improvement in decoding throughput.

0 citationsRead paper