Selected work

Representative Papers

Key-Value Means

May 10, 2026

This work addresses the high computational complexity, substantial memory footprint, and poor state scalability of conventional attention mechanisms in long-context scenarios by introducing Key-Value Means (KVM), a novel block-recurrent attention mechanism. KVM constructs a recurrent structure using standard attention operations—without requiring custom kernels—and supports both fixed and growable state representations while preserving parallel training capability and enabling linear-time inference. Its key contribution lies in unifying the scalable memory of Transformers with the efficient inference of linear RNNs, allowing continuous adjustment of prefill complexity from O(N) to O(N²). This significantly reduces KV cache overhead and achieves sub-quadratic prefill time and sub-linear state growth on long-context tasks, all with minimal additional parameters. The code and models are publicly released.

0 citationsRead paper

Matrix-Driven Instant Review: Confident Detection and Reconstruction of LLM Plagiarism on PC

Aug 08, 2025

Existing LLM weight plagiarism detection methods suffer from inaccurate weight mapping reconstruction, lack of statistically rigorous significance measures (e.g., *p*-values), and high false-positive rates due to training data similarity. Method: This paper proposes a novel detection framework grounded in matrix analysis and large deviations theory. It integrates spectral alignment with matrix similarity analysis—requiring neither model inference nor full weight loading—and reconstructs cross-model weight relationships interpretably using only inter-layer weight matrix structure, while providing theoretically grounded statistical significance assessment (*p*-values). Contributions/Results: Experiments demonstrate strong robustness against severe perturbations—including random weight permutation, structured pruning, and trillion-token-scale continued pretraining—while achieving sub-hour full-model detection on a single PC. The method significantly outperforms state-of-the-art approaches in both accuracy and reliability.

0 citationsRead paper

RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale

May 05, 2025

To address the high inference overhead and deployment challenges of softmax-based attention in Transformers, this paper proposes an efficient distillation framework that rapidly converts standard Transformer models (7B/32B/72B) into linear-complexity RWKV-style decoders. The method integrates attention mechanism distillation, structural reparameterization, and lightweight fine-tuning, requiring only 350–700M tokens (<0.005% of original pretraining data) for conversion. It marks the first successful realization of a 72B-scale linear-attention decoder, achieving inference quality nearly on par with the original models and attaining state-of-the-art performance among comparable methods on mainstream benchmarks. The total conversion cost remains under $2,000. All distilled models are publicly released under the Apache 2.0 + Qwen license, enabling broad research and practical adoption.

0 citationsRead paper

RWKV-7"Goose"with Expressive Dynamic State Evolution

Mar 18, 2025

This work addresses two fundamental limitations in sequence modeling: the difficulty of simultaneously achieving parallel training and efficient state tracking, and the theoretical restriction of Transformers to the complexity class $mathsf{TC}^0$—rendering them incapable of recognizing all regular languages. To overcome these, we propose RWKV-7 “Goose”, a novel architecture featuring: (1) a generalized delta rule with vector-valued gating and context-aware learning rates, coupled with a relaxed value replacement mechanism; (2) recurrent state evolution with constant memory and inference overhead; and (3) the first formal proof that the architecture can recognize all regular languages—surpassing Transformer’s $mathsf{TC}^0$ bound. Trained on a proprietary 3.1-trillion-token multilingual corpus, the 3B-parameter model achieves new SOTA in multilingual understanding and generation, with significantly reduced training token requirements. We open-source four models (0.19B–2.9B parameters) and full code under Apache 2.0, available on Hugging Face and GitHub.

0 citationsRead paper

Recent publications

Latest Papers

Key-Value Means

May 10, 2026

This work addresses the high computational complexity, substantial memory footprint, and poor state scalability of conventional attention mechanisms in long-context scenarios by introducing Key-Value Means (KVM), a novel block-recurrent attention mechanism. KVM constructs a recurrent structure using standard attention operations—without requiring custom kernels—and supports both fixed and growable state representations while preserving parallel training capability and enabling linear-time inference. Its key contribution lies in unifying the scalable memory of Transformers with the efficient inference of linear RNNs, allowing continuous adjustment of prefill complexity from O(N) to O(N²). This significantly reduces KV cache overhead and achieves sub-quadratic prefill time and sub-linear state growth on long-context tasks, all with minimal additional parameters. The code and models are publicly released.

0 citationsRead paper

Matrix-Driven Instant Review: Confident Detection and Reconstruction of LLM Plagiarism on PC

Aug 08, 2025

Existing LLM weight plagiarism detection methods suffer from inaccurate weight mapping reconstruction, lack of statistically rigorous significance measures (e.g., *p*-values), and high false-positive rates due to training data similarity. Method: This paper proposes a novel detection framework grounded in matrix analysis and large deviations theory. It integrates spectral alignment with matrix similarity analysis—requiring neither model inference nor full weight loading—and reconstructs cross-model weight relationships interpretably using only inter-layer weight matrix structure, while providing theoretically grounded statistical significance assessment (*p*-values). Contributions/Results: Experiments demonstrate strong robustness against severe perturbations—including random weight permutation, structured pruning, and trillion-token-scale continued pretraining—while achieving sub-hour full-model detection on a single PC. The method significantly outperforms state-of-the-art approaches in both accuracy and reliability.

0 citationsRead paper

RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale

May 05, 2025

To address the high inference overhead and deployment challenges of softmax-based attention in Transformers, this paper proposes an efficient distillation framework that rapidly converts standard Transformer models (7B/32B/72B) into linear-complexity RWKV-style decoders. The method integrates attention mechanism distillation, structural reparameterization, and lightweight fine-tuning, requiring only 350–700M tokens (<0.005% of original pretraining data) for conversion. It marks the first successful realization of a 72B-scale linear-attention decoder, achieving inference quality nearly on par with the original models and attaining state-of-the-art performance among comparable methods on mainstream benchmarks. The total conversion cost remains under $2,000. All distilled models are publicly released under the Apache 2.0 + Qwen license, enabling broad research and practical adoption.

0 citationsRead paper

RWKV-7"Goose"with Expressive Dynamic State Evolution

Mar 18, 2025

This work addresses two fundamental limitations in sequence modeling: the difficulty of simultaneously achieving parallel training and efficient state tracking, and the theoretical restriction of Transformers to the complexity class $mathsf{TC}^0$—rendering them incapable of recognizing all regular languages. To overcome these, we propose RWKV-7 “Goose”, a novel architecture featuring: (1) a generalized delta rule with vector-valued gating and context-aware learning rates, coupled with a relaxed value replacement mechanism; (2) recurrent state evolution with constant memory and inference overhead; and (3) the first formal proof that the architecture can recognize all regular languages—surpassing Transformer’s $mathsf{TC}^0$ bound. Trained on a proprietary 3.1-trillion-token multilingual corpus, the 3B-parameter model achieves new SOTA in multilingual understanding and generation, with significantly reduced training token requirements. We open-source four models (0.19B–2.9B parameters) and full code under Apache 2.0, available on Hugging Face and GitHub.

0 citationsRead paper