Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
为解决长上下文输入时响应质量下降和时间延迟问题，提出结合KV缓存拼接意识微调模型与选择性重计算KV缓存的方法。
0 citationsRead paper
Selected work
Representative Papers
Top-K Retrieval with Fixed-Size Linear-Attention Completion: Backbone- and KV-Format-Preserving Attention for KV-Cache Read Reduction
This work addresses the decoding bandwidth bottleneck in long-context generation caused by frequent reads of the key-value (KV) cache, which becomes especially severe when the cache is offloaded to GPU external memory. The authors propose a retrieval-completion attention mechanism that, without altering the model architecture or KV cache format, computes exact attention over the top-K tokens most relevant to the query along with fixed anchor tokens at the sequence boundaries. Contributions from intermediate regions are approximated using a fixed-size feature map constructed during the prefill phase, and a unified normalization recovers the full attention distribution. This approach effectively mitigates the softmax bias introduced by top-K retrieval while requiring no additional KV cache accesses. Under identical KV access budgets, it significantly outperforms pure top-K methods, with particularly notable gains in high-entropy attention heads.
0 citationsRead paper
Revolutionizing Long-Term Memory in AI: New Horizons with High-Capacity and High-Speed Storage
Current AI systems commonly adopt a “extract-then-store” memory paradigm, which often discards task-irrelevant yet potentially useful information, thereby limiting the development of long-term memory capabilities. This work proposes a novel “store-then-extract-on-demand” paradigm that preserves all raw experiences in full, enabling flexible retrieval across multiple tasks. By incorporating mechanisms for aggregative analysis based on large-scale probabilistic experiences and cross-task experience sharing, the approach enhances memory utility without relying on new learning algorithms—instead, it reconfigures the memory system architecture itself. Preliminary experiments demonstrate significant improvements in adaptability, reuse efficiency, and knowledge discovery within the memory system, offering a promising pathway toward high-capacity, high-throughput long-term memory architectures for artificial intelligence.
0 citationsRead paper
Analysis and Evaluation of Using Microsecond-Latency Memory for In-Memory Indices and Caches in SSD-Based Key-Value Stores
SSD-based key-value (KV) stores heavily rely on expensive host DRAM for indexing and caching, creating a cost bottleneck. This work investigates migrating these data structures to lower-cost secondary memory with microsecond-scale latency (1–5 μs), but faces the challenge of sustaining high KV throughput under such latency. Method: We propose a user-thread–based software prefetching mechanism that explicitly models and jointly optimizes prefetching and I/O scheduling—without requiring new hardware or protocol modifications—to effectively hide secondary memory latency. Contribution/Results: Using an FPGA-based tunable-latency memory platform, we validate our approach via microbenchmarks and a modified KV store. Results show near-DRAM throughput is maintained even at up to 5 μs memory latency. To our knowledge, this is the first work to demonstrate that microsecond-latency secondary memory can serve as a cost-effective, drop-in DRAM alternative—achieving this without custom hardware or modifications to the storage stack.
0 citationsRead paper
Prompt-Guided Attention Head Selection for Focus-Oriented Image Retrieval
This work addresses the challenge in focus-oriented image retrieval (FOIR) under multi-object complex scenes, where pretrained Vision Transformers (ViTs) struggle to precisely respond to user-provided visual prompts (e.g., points, bounding boxes, or masks). We propose a prompt-driven attention head selection mechanism that requires no fine-tuning and modifies neither model weights nor input images. At inference time, it dynamically matches each visual prompt against attention maps generated by individual heads, activating those most sensitive to the target region—thereby jointly capturing local object details and contextual information. As the first zero-shot, training-agnostic, plug-and-play ViT adaptation for FOIR, our method significantly improves retrieval accuracy across multiple benchmarks while preserving the original ViT’s inference efficiency. Extensive experiments validate its effectiveness, broad applicability across ViT variants, and deployment friendliness.
0 citationsRead paper
Recent publications
Latest Papers
Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
为解决长上下文输入时响应质量下降和时间延迟问题，提出结合KV缓存拼接意识微调模型与选择性重计算KV缓存的方法。
0 citationsRead paper
Top-K Retrieval with Fixed-Size Linear-Attention Completion: Backbone- and KV-Format-Preserving Attention for KV-Cache Read Reduction
This work addresses the decoding bandwidth bottleneck in long-context generation caused by frequent reads of the key-value (KV) cache, which becomes especially severe when the cache is offloaded to GPU external memory. The authors propose a retrieval-completion attention mechanism that, without altering the model architecture or KV cache format, computes exact attention over the top-K tokens most relevant to the query along with fixed anchor tokens at the sequence boundaries. Contributions from intermediate regions are approximated using a fixed-size feature map constructed during the prefill phase, and a unified normalization recovers the full attention distribution. This approach effectively mitigates the softmax bias introduced by top-K retrieval while requiring no additional KV cache accesses. Under identical KV access budgets, it significantly outperforms pure top-K methods, with particularly notable gains in high-entropy attention heads.
0 citationsRead paper
Revolutionizing Long-Term Memory in AI: New Horizons with High-Capacity and High-Speed Storage
Current AI systems commonly adopt a “extract-then-store” memory paradigm, which often discards task-irrelevant yet potentially useful information, thereby limiting the development of long-term memory capabilities. This work proposes a novel “store-then-extract-on-demand” paradigm that preserves all raw experiences in full, enabling flexible retrieval across multiple tasks. By incorporating mechanisms for aggregative analysis based on large-scale probabilistic experiences and cross-task experience sharing, the approach enhances memory utility without relying on new learning algorithms—instead, it reconfigures the memory system architecture itself. Preliminary experiments demonstrate significant improvements in adaptability, reuse efficiency, and knowledge discovery within the memory system, offering a promising pathway toward high-capacity, high-throughput long-term memory architectures for artificial intelligence.
0 citationsRead paper
Analysis and Evaluation of Using Microsecond-Latency Memory for In-Memory Indices and Caches in SSD-Based Key-Value Stores
SSD-based key-value (KV) stores heavily rely on expensive host DRAM for indexing and caching, creating a cost bottleneck. This work investigates migrating these data structures to lower-cost secondary memory with microsecond-scale latency (1–5 μs), but faces the challenge of sustaining high KV throughput under such latency. Method: We propose a user-thread–based software prefetching mechanism that explicitly models and jointly optimizes prefetching and I/O scheduling—without requiring new hardware or protocol modifications—to effectively hide secondary memory latency. Contribution/Results: Using an FPGA-based tunable-latency memory platform, we validate our approach via microbenchmarks and a modified KV store. Results show near-DRAM throughput is maintained even at up to 5 μs memory latency. To our knowledge, this is the first work to demonstrate that microsecond-latency secondary memory can serve as a cost-effective, drop-in DRAM alternative—achieving this without custom hardware or modifications to the storage stack.
0 citationsRead paper
Prompt-Guided Attention Head Selection for Focus-Oriented Image Retrieval
This work addresses the challenge in focus-oriented image retrieval (FOIR) under multi-object complex scenes, where pretrained Vision Transformers (ViTs) struggle to precisely respond to user-provided visual prompts (e.g., points, bounding boxes, or masks). We propose a prompt-driven attention head selection mechanism that requires no fine-tuning and modifies neither model weights nor input images. At inference time, it dynamically matches each visual prompt against attention maps generated by individual heads, activating those most sensitive to the target region—thereby jointly capturing local object details and contextual information. As the first zero-shot, training-agnostic, plug-and-play ViT adaptation for FOIR, our method significantly improves retrieval accuracy across multiple benchmarks while preserving the original ViT’s inference efficiency. Extensive experiments validate its effectiveness, broad applicability across ViT variants, and deployment friendliness.
0 citationsRead paper