Selected work

Representative Papers

TRADE: Transducer-Augmented Decoder for Speech LLM

Jun 07, 2026

This work addresses the challenge that existing speech large language models lack streaming inference mechanisms, making it difficult to simultaneously achieve real-time decoding and accurate end-of-utterance detection. The authors propose a tightly coupled dual-vocabulary architecture that integrates frame-synchronous acoustic alignment with language model reasoning through a shared audio encoder and transformer branches. The method incorporates chunk-synchronous streaming training, gradient truncation, localized audio attention, causal sliding windows, and zero-overhead score fusion to enable efficient streaming recognition and support for long-form audio. Evaluated on the Open ASR Leaderboard, the approach achieves an average word error rate (WER) of 6.71%, with 8.40% WER under 960ms chunked streaming conditions. It also attains 3.64% and 10.88% WER on TED-LIUM and Earnings-22, respectively, while improving the end-of-utterance detection F₁ score by 0.03.

0 citationsRead paper

A Simple Plug-in for Improving Eviction-Based KV Cache Compression

May 22, 2026

This work addresses the memory bottleneck of key-value (KV) caching in large language models during long-context reasoning, where existing methods struggle to balance information retention and compression efficiency due to binary eviction or coarse approximation strategies. The authors propose VECTOR, a plug-and-play KV cache enhancement module that introduces a novel three-way token routing mechanism—preserving, reconstructibly approximating, or evicting tokens—by integrating importance scores with offline-calibrated reconstructability signals. This enables finer-grained cache management and effectively recovers critical information permanently lost under conventional binary strategies. Empirical results demonstrate that VECTOR significantly improves the quality-memory trade-off at moderate to high compression ratios, with particularly pronounced gains under stringent memory constraints.

0 citationsRead paper

Crafting Reversible SFT Behaviors in Large Language Models

May 07, 2026

This work addresses the lack of structural constraints in behaviors induced by supervised fine-tuning (SFT), which renders them difficult to controllably activate or reverse during inference. The authors propose compressing SFT-induced behaviors into sparse, mechanistically necessary subnetworks—termed carriers—that serve as causally necessary, rather than merely correlational, substrates for target behaviors. They demonstrate that such sparsity is a critical prerequisite for behavioral reversibility. By jointly optimizing routing masks and model weights via Loss-Constrained Double Descent (LCDD) and integrating activation-matching-based soft prompts through SFT-Eraser, the method enables precise on-demand triggering or reversal of target behaviors at inference time without modifying model weights. Experiments across diverse models and behavior types confirm that this approach effectively preserves and robustly reverses target behaviors, a capability unattainable with standard SFT.

0 citationsRead paper

RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts

Apr 28, 2026

Existing MoE inference schedulers rely solely on batch size while ignoring expert routing distributions, resulting in 10%–70% kernel throughput waste. This work proposes RaMP, a novel framework that introduces the first routing-aware kernel scheduling mechanism. RaMP constructs a kernel-agnostic four-parameter wave cost model using runtime expert histograms and lightweight one-time profiling (10–24 minutes), then dynamically selects optimal kernel configurations by integrating CTA grid geometry modeling with performance region analysis. The approach accurately predicts performance regions even on unseen architectures. Experimental results demonstrate that RaMP achieves a 1.22× speedup over static scheduling at the kernel level. When integrated into vLLM, it outperforms state-of-the-art backends—delivering end-to-end inference speedups of 1.30× over Triton, 1.41× over DeepGEMM, and 1.13× over FlashInfer CUTLASS.

0 citationsRead paper

ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detection

Feb 09, 2026

This work addresses the challenge of linearly growing key-value (KV) cache memory with sequence length in long-context reasoning, which necessitates efficient compression to preserve critical historical information. The authors propose a training-free, plug-and-play KV cache compression method that, for the first time, employs Euclidean distance—rather than cosine similarity—to measure the deviation of key vectors from local centroids, thereby jointly capturing both angular and magnitude information to assess token importance. This approach effectively mitigates directional conflicts and global centroid dilution. A sliding window mechanism is further introduced to enhance robustness in ultra-long contexts. On the RULER benchmark, the method achieves 95.7% accuracy at 20% compression across 4K–16K contexts, 92.4% accuracy under 50% compression on the 3-key NIAH task (a 15.4-point improvement over the baseline), and 84.3% accuracy at 25% compression in 64K contexts, recovering performance by a remarkable 49 points.

0 citationsRead paper

Recent publications

Latest Papers

TRADE: Transducer-Augmented Decoder for Speech LLM

Jun 07, 2026

This work addresses the challenge that existing speech large language models lack streaming inference mechanisms, making it difficult to simultaneously achieve real-time decoding and accurate end-of-utterance detection. The authors propose a tightly coupled dual-vocabulary architecture that integrates frame-synchronous acoustic alignment with language model reasoning through a shared audio encoder and transformer branches. The method incorporates chunk-synchronous streaming training, gradient truncation, localized audio attention, causal sliding windows, and zero-overhead score fusion to enable efficient streaming recognition and support for long-form audio. Evaluated on the Open ASR Leaderboard, the approach achieves an average word error rate (WER) of 6.71%, with 8.40% WER under 960ms chunked streaming conditions. It also attains 3.64% and 10.88% WER on TED-LIUM and Earnings-22, respectively, while improving the end-of-utterance detection F₁ score by 0.03.

0 citationsRead paper

A Simple Plug-in for Improving Eviction-Based KV Cache Compression

May 22, 2026

This work addresses the memory bottleneck of key-value (KV) caching in large language models during long-context reasoning, where existing methods struggle to balance information retention and compression efficiency due to binary eviction or coarse approximation strategies. The authors propose VECTOR, a plug-and-play KV cache enhancement module that introduces a novel three-way token routing mechanism—preserving, reconstructibly approximating, or evicting tokens—by integrating importance scores with offline-calibrated reconstructability signals. This enables finer-grained cache management and effectively recovers critical information permanently lost under conventional binary strategies. Empirical results demonstrate that VECTOR significantly improves the quality-memory trade-off at moderate to high compression ratios, with particularly pronounced gains under stringent memory constraints.

0 citationsRead paper

Crafting Reversible SFT Behaviors in Large Language Models

May 07, 2026

This work addresses the lack of structural constraints in behaviors induced by supervised fine-tuning (SFT), which renders them difficult to controllably activate or reverse during inference. The authors propose compressing SFT-induced behaviors into sparse, mechanistically necessary subnetworks—termed carriers—that serve as causally necessary, rather than merely correlational, substrates for target behaviors. They demonstrate that such sparsity is a critical prerequisite for behavioral reversibility. By jointly optimizing routing masks and model weights via Loss-Constrained Double Descent (LCDD) and integrating activation-matching-based soft prompts through SFT-Eraser, the method enables precise on-demand triggering or reversal of target behaviors at inference time without modifying model weights. Experiments across diverse models and behavior types confirm that this approach effectively preserves and robustly reverses target behaviors, a capability unattainable with standard SFT.

0 citationsRead paper

RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts

Apr 28, 2026

Existing MoE inference schedulers rely solely on batch size while ignoring expert routing distributions, resulting in 10%–70% kernel throughput waste. This work proposes RaMP, a novel framework that introduces the first routing-aware kernel scheduling mechanism. RaMP constructs a kernel-agnostic four-parameter wave cost model using runtime expert histograms and lightweight one-time profiling (10–24 minutes), then dynamically selects optimal kernel configurations by integrating CTA grid geometry modeling with performance region analysis. The approach accurately predicts performance regions even on unseen architectures. Experimental results demonstrate that RaMP achieves a 1.22× speedup over static scheduling at the kernel level. When integrated into vLLM, it outperforms state-of-the-art backends—delivering end-to-end inference speedups of 1.30× over Triton, 1.41× over DeepGEMM, and 1.13× over FlashInfer CUTLASS.

0 citationsRead paper

ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detection

Feb 09, 2026

This work addresses the challenge of linearly growing key-value (KV) cache memory with sequence length in long-context reasoning, which necessitates efficient compression to preserve critical historical information. The authors propose a training-free, plug-and-play KV cache compression method that, for the first time, employs Euclidean distance—rather than cosine similarity—to measure the deviation of key vectors from local centroids, thereby jointly capturing both angular and magnitude information to assess token importance. This approach effectively mitigates directional conflicts and global centroid dilution. A sliding window mechanism is further introduced to enhance robustness in ultra-long contexts. On the RULER benchmark, the method achieves 95.7% accuracy at 20% compression across 4K–16K contexts, 92.4% accuracy under 50% compression on the 3-key NIAH task (a 15.4-point improvement over the baseline), and 84.3% accuracy at 25% compression in 64K contexts, recovering performance by a remarkable 49 points.

0 citationsRead paper