Selected work

Representative Papers

Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion

Apr 07, 2026

This work addresses the limitations of existing large language models in long-context and long-generation tasks, which stem from the substantial memory and bandwidth overhead of key-value (KV) cache in attention mechanisms. While more efficient attention architectures have been proposed, they are difficult to integrate into already-trained models without costly retraining. To overcome this, the authors introduce an Attention Editing framework that replaces the original attention mechanism with a learnable target module and employs a progressive distillation strategy to enable cross-architecture transfer—eliminating the need for full pretraining. This approach relaxes prior fine-grained structural constraints between source and target architectures and achieves, for the first time, a general and practical method for attention replacement in large-scale models. The framework successfully deploys MLA and GateSWA on Qwen3-8B and Qwen3-30B-A3B, significantly boosting inference efficiency on Ascend 910B clusters while preserving model performance.

0 citationsRead paper

ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models

Aug 25, 2025

Large language models (LLMs) face fundamental scalability challenges with ultra-long contexts (e.g., 1M tokens), owing to quadratic computational complexity, excessive memory consumption, and limited effective context length. To address this, we propose Intermediate-Layer Retrieval (ILRe): an offline-selected decoder intermediate layer is leveraged for streaming chunked prefilling, combined with attention-score-driven key-token recall. We further introduce a novel multi-pooling kernel allocation strategy that reduces prefilling complexity from $O(L^2)$ to $O(L)$, without requiring post-training or custom operators. Evaluated on the Llama-3.1-UltraLong-8B-1M-Instruct model deployed on Ascend 910B, ILRe processes a single 1M-token request in under 30 seconds—achieving ~180× inference speedup over baseline methods—while attaining a RULER-1M score of 79.8, matching full-context performance.

0 citationsRead paper

Lag-Relative Sparse Attention In Long Context Training

Jun 13, 2025

Large language models (LLMs) face dual bottlenecks in long-context modeling: quadratic attention complexity (O(n²)) and linear memory growth of key-value (KV) caches. Existing KV compression methods, designed post-hoc without training-time integration, incur substantial inference degradation and poor compatibility with post-training workflows. To address this, we propose Lagged Relative Sparse Attention (LRSA), the first method to embed the LagKV compression mechanism directly into model training—enabling parameter-free, gradient-differentiable, and low-overhead sparsification. LRSA introduces a lagged window Top-K selection strategy and block-wise pre-filling to balance efficiency and robustness. Experiments on question-answering fine-tuning show LRSA significantly outperforms baselines under compressed contexts, with near-zero performance loss, negligible training overhead, and seamless compatibility with both end-to-end training and post-training—marking the first KV compression approach fully integrated into standard LLM optimization pipelines.

0 citationsRead paper

LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important

Apr 07, 2025

To address the deployment cost–accuracy trade-off caused by KV cache explosion in long-context reasoning with large language models, this paper proposes a lightweight, attention-free KV cache compression method. The core innovation is a novel importance criterion based on temporal lag relationships between key-value pairs, enabling gradient-free, non-intrusive cache importance estimation via purely comparative relative lag analysis and autoregressive local sensitivity modeling. Crucially, the method requires no modifications to the inference framework and incurs negligible computational overhead. On LongBench and PasskeyRetrieval benchmarks, 2× compression achieves near-lossless performance, while 8× compression retains approximately 90% of original accuracy. In 64-bit key retrieval tasks, it outperforms H₂O by over 60% in accuracy.

0 citationsRead paper

Rewarding Curse: Analyze and Mitigate Reward Modeling Issues for LLM Reasoning

Mar 07, 2025

Large language models (LLMs) exhibit inconsistency between effectiveness and faithfulness in chain-of-thought (CoT) reasoning, primarily due to biases in reward modeling. Method: We propose the first holistic analytical framework jointly modeling the “question–CoT–answer” information interaction, leveraging information quantification, faithfulness diagnosis, and information gain assessment to uncover how question difficulty, information gain, and directional information flow critically influence CoT performance. We identify a prevalent failure mode wherein models bypass incomplete CoTs and directly retrieve answers from the question—yielding superficially correct but logically unfaithful outputs. Building on these insights, we design a novel CoT generation and evaluation algorithm driven by question-informed information backtracking and information gain optimization. Contribution/Results: Evaluated across diverse reasoning tasks, our approach significantly improves CoT faithfulness (+21.3%) and effectiveness (+14.7% accuracy), empirically validating the centrality of information backtracking in faithful reasoning.

0 citationsRead paper

Recent publications

Latest Papers

Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion

Apr 07, 2026

This work addresses the limitations of existing large language models in long-context and long-generation tasks, which stem from the substantial memory and bandwidth overhead of key-value (KV) cache in attention mechanisms. While more efficient attention architectures have been proposed, they are difficult to integrate into already-trained models without costly retraining. To overcome this, the authors introduce an Attention Editing framework that replaces the original attention mechanism with a learnable target module and employs a progressive distillation strategy to enable cross-architecture transfer—eliminating the need for full pretraining. This approach relaxes prior fine-grained structural constraints between source and target architectures and achieves, for the first time, a general and practical method for attention replacement in large-scale models. The framework successfully deploys MLA and GateSWA on Qwen3-8B and Qwen3-30B-A3B, significantly boosting inference efficiency on Ascend 910B clusters while preserving model performance.

0 citationsRead paper

ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models

Aug 25, 2025

Large language models (LLMs) face fundamental scalability challenges with ultra-long contexts (e.g., 1M tokens), owing to quadratic computational complexity, excessive memory consumption, and limited effective context length. To address this, we propose Intermediate-Layer Retrieval (ILRe): an offline-selected decoder intermediate layer is leveraged for streaming chunked prefilling, combined with attention-score-driven key-token recall. We further introduce a novel multi-pooling kernel allocation strategy that reduces prefilling complexity from $O(L^2)$ to $O(L)$, without requiring post-training or custom operators. Evaluated on the Llama-3.1-UltraLong-8B-1M-Instruct model deployed on Ascend 910B, ILRe processes a single 1M-token request in under 30 seconds—achieving ~180× inference speedup over baseline methods—while attaining a RULER-1M score of 79.8, matching full-context performance.

0 citationsRead paper

Lag-Relative Sparse Attention In Long Context Training

Jun 13, 2025

Large language models (LLMs) face dual bottlenecks in long-context modeling: quadratic attention complexity (O(n²)) and linear memory growth of key-value (KV) caches. Existing KV compression methods, designed post-hoc without training-time integration, incur substantial inference degradation and poor compatibility with post-training workflows. To address this, we propose Lagged Relative Sparse Attention (LRSA), the first method to embed the LagKV compression mechanism directly into model training—enabling parameter-free, gradient-differentiable, and low-overhead sparsification. LRSA introduces a lagged window Top-K selection strategy and block-wise pre-filling to balance efficiency and robustness. Experiments on question-answering fine-tuning show LRSA significantly outperforms baselines under compressed contexts, with near-zero performance loss, negligible training overhead, and seamless compatibility with both end-to-end training and post-training—marking the first KV compression approach fully integrated into standard LLM optimization pipelines.

0 citationsRead paper

LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important

Apr 07, 2025

To address the deployment cost–accuracy trade-off caused by KV cache explosion in long-context reasoning with large language models, this paper proposes a lightweight, attention-free KV cache compression method. The core innovation is a novel importance criterion based on temporal lag relationships between key-value pairs, enabling gradient-free, non-intrusive cache importance estimation via purely comparative relative lag analysis and autoregressive local sensitivity modeling. Crucially, the method requires no modifications to the inference framework and incurs negligible computational overhead. On LongBench and PasskeyRetrieval benchmarks, 2× compression achieves near-lossless performance, while 8× compression retains approximately 90% of original accuracy. In 64-bit key retrieval tasks, it outperforms H₂O by over 60% in accuracy.

0 citationsRead paper

Rewarding Curse: Analyze and Mitigate Reward Modeling Issues for LLM Reasoning

Mar 07, 2025

Large language models (LLMs) exhibit inconsistency between effectiveness and faithfulness in chain-of-thought (CoT) reasoning, primarily due to biases in reward modeling. Method: We propose the first holistic analytical framework jointly modeling the “question–CoT–answer” information interaction, leveraging information quantification, faithfulness diagnosis, and information gain assessment to uncover how question difficulty, information gain, and directional information flow critically influence CoT performance. We identify a prevalent failure mode wherein models bypass incomplete CoTs and directly retrieve answers from the question—yielding superficially correct but logically unfaithful outputs. Building on these insights, we design a novel CoT generation and evaluation algorithm driven by question-informed information backtracking and information gain optimization. Contribution/Results: Evaluated across diverse reasoning tasks, our approach significantly improves CoT faithfulness (+21.3%) and effectiveness (+14.7% accuracy), empirically validating the centrality of information backtracking in faithful reasoning.

0 citationsRead paper