DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing

2026-04-21

AI Summary

This work addresses the O(N²) computational bottleneck in long-context reasoning with large language models caused by standard attention mechanisms. The authors propose an asymmetric deep hashing-based approximate nearest neighbor attention mechanism that differentially encodes queries and keys, reframing attention computation as a hashing retrieval problem for the first time. By integrating a dynamic mixed-precision strategy—preserving full precision for critical tokens while using low-precision representations elsewhere—the method balances efficiency and generation quality. The resulting asymmetric KV-cache hashing framework substantially reduces memory and computational overhead, achieving performance on par with full attention on the LongBench benchmark while lowering inference complexity to O(N).

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The quadratic computational complexity of the standard attention mechanism constitutes a fundamental bottleneck for large language models in long-context inference. While existing KV cache compression methods alleviate memory pressure, they often sacrifice generation quality and fail to address the high overhead of floating-point arithmetic. This paper introduces DASH-KV, an innovative acceleration framework that reformulates attention as approximate nearest-neighbor search via asymmetric deep hashing. Under this paradigm, we design an asymmetric encoding architecture that differentially maps queries and keys to account for their distinctions in precision and reuse characteristics. To balance efficiency and accuracy, we further introduce a dynamic mixed-precision mechanism that adaptively retains full-precision computation for critical tokens. Extensive experiments on LongBench demonstrate that DASH-KV significantly outperforms state-of-the-art baseline methods while matching the performance of full attention, all while reducing inference complexity from O(N^2) to linear O(N). The code is available at https://github.com/Zhihan-Zh/DASH-KV

Problem

• long-context inference
• attention mechanism
• KV cache
• computational complexity
• floating-point arithmetic

Innovation

• asymmetric hashing
• KV cache compression
• approximate nearest-neighbor search
• dynamic mixed-precision
• linear attention complexity