Selected work

Representative Papers

Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection

Aug 03, 2026

This work addresses the high latency of conventional Retrieval-Augmented Generation (RAG) on edge devices, caused by context prefilling and KV cache overhead, which impedes real-time interaction. The authors propose PRECOG, a mechanism integrating Structured Memory Consolidation (SMC) with state space models (SSMs), leveraging their fixed-size hidden states to encode document corpora offline and inject the optimal matching state directly at query time. This approach achieves O(1) complexity for zero-context reinjection and enables hierarchical persistent memory fusion, circumventing the positional entanglement limitations inherent in Transformers. Evaluated on a 1.2B-parameter TENNs-LLM, the method reduces prefill latency from 27 seconds to under 6 milliseconds—a speedup of approximately 4,500×—enabling real-time edge interaction while preserving answer quality comparable to traditional RAG.

0 citationsRead paper

A New Kind of Adversarial Example: Measuring the Human-Model Gap, and Its Relationship to OOD Detection

Jul 21, 2026

This study reveals a human–model cognitive gap: when subjected to visibly perceptible perturbations, human recognition accuracy drops substantially (by approximately 49%), whereas deep models maintain near-perfect prediction accuracy. Through large-scale human experiments, CLIP zero-shot proxies, and Gaussian-controlled perturbation generation, the work systematically evaluates the effectiveness of mainstream out-of-distribution (OOD) detection, calibration, and defense methods. It provides the first empirical validation of this performance disparity, demonstrating that existing OOD detectors largely fail—only Mahalanobis distance shows partial efficacy, yet remains vulnerable to adaptive attacks—and that conventional defense strategies are ineffective. The analysis further indicates that such perturbations primarily disrupt texture rather than structural features.

0 citationsRead paper

Do Transformers Need Three Projections? Systematic Study of QKV Variants

Jun 01, 2026

This work investigates the necessity of distinct query (Q), key (K), and value (V) projections in Transformer models and proposes three parameter-sharing strategies: Q–K=V, Q=K–V, and Q=K=V. To mitigate symmetric attention patterns arising from shared projections, the authors introduce a two-dimensional positional encoding scheme. Empirical results demonstrate that the Q–K=V configuration incurs only a 3.1% increase in perplexity on language modeling tasks while reducing KV cache memory by 50%. When combined with multi-query or grouped-query attention (MQA/GQA), this approach achieves up to 96.9% KV cache compression, maintaining competitive or superior performance across both vision and language benchmarks. The method substantially lowers memory overhead, seamlessly integrates with existing head-sharing mechanisms, and offers a promising pathway for efficient on-device deployment.

0 citationsRead paper

Provenance Networks: End-to-End Exemplar-Based Explainability

Oct 02, 2025

Deep learning models suffer from poor interpretability, frequent hallucinations, and difficulty tracing predictions back to training samples—undermining their robustness and trustworthiness. To address this, we propose TraceNet, an architecture that intrinsically embeds example-based interpretability via end-to-end prediction-to-sample association. Our core innovation is a learnable weighted k-nearest neighbors (k-NN) mechanism that dynamically retrieves and weights supportive training instances in the feature space, jointly optimizing both the primary task loss and an explicit interpretability objective. TraceNet enables training-data provenance, label-noise detection, improved robustness to input perturbations, and explicit generation grounding. Experiments on medium-scale benchmarks demonstrate substantial gains in decision transparency and yield novel insights into the interplay between memorization and generalization.

0 citationsRead paper

Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions

Jan 22, 2025

State space models (SSMs) underperform in audio tasks due to limited modeling capacity, high computational overhead, and lack of inductive bias from convolutional priors. Method: We propose Centaurus—a fully SSM-based architecture that reformulates generalized SSMs as learnable tensor contraction operations with automatically optimized contraction orders; it is the first to systematically integrate group convolution, full convolution, and bottleneck structures into a heterogeneous SSM design. Contribution/Results: Centaurus is the first high-performance automatic speech recognition (ASR) model built entirely upon SSMs—without LSTM, explicit CNNs, or attention mechanisms. It outperforms homogeneous SSM baselines of comparable size on keyword spotting, speech denoising, and ASR. On ASR, it achieves state-of-the-art (SOTA) performance while significantly reducing memory consumption and computational cost during both training and inference.

0 citationsRead paper

Recent publications

Latest Papers

Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection

Aug 03, 2026

This work addresses the high latency of conventional Retrieval-Augmented Generation (RAG) on edge devices, caused by context prefilling and KV cache overhead, which impedes real-time interaction. The authors propose PRECOG, a mechanism integrating Structured Memory Consolidation (SMC) with state space models (SSMs), leveraging their fixed-size hidden states to encode document corpora offline and inject the optimal matching state directly at query time. This approach achieves O(1) complexity for zero-context reinjection and enables hierarchical persistent memory fusion, circumventing the positional entanglement limitations inherent in Transformers. Evaluated on a 1.2B-parameter TENNs-LLM, the method reduces prefill latency from 27 seconds to under 6 milliseconds—a speedup of approximately 4,500×—enabling real-time edge interaction while preserving answer quality comparable to traditional RAG.

0 citationsRead paper

A New Kind of Adversarial Example: Measuring the Human-Model Gap, and Its Relationship to OOD Detection

Jul 21, 2026

This study reveals a human–model cognitive gap: when subjected to visibly perceptible perturbations, human recognition accuracy drops substantially (by approximately 49%), whereas deep models maintain near-perfect prediction accuracy. Through large-scale human experiments, CLIP zero-shot proxies, and Gaussian-controlled perturbation generation, the work systematically evaluates the effectiveness of mainstream out-of-distribution (OOD) detection, calibration, and defense methods. It provides the first empirical validation of this performance disparity, demonstrating that existing OOD detectors largely fail—only Mahalanobis distance shows partial efficacy, yet remains vulnerable to adaptive attacks—and that conventional defense strategies are ineffective. The analysis further indicates that such perturbations primarily disrupt texture rather than structural features.

0 citationsRead paper

Do Transformers Need Three Projections? Systematic Study of QKV Variants

Jun 01, 2026

This work investigates the necessity of distinct query (Q), key (K), and value (V) projections in Transformer models and proposes three parameter-sharing strategies: Q–K=V, Q=K–V, and Q=K=V. To mitigate symmetric attention patterns arising from shared projections, the authors introduce a two-dimensional positional encoding scheme. Empirical results demonstrate that the Q–K=V configuration incurs only a 3.1% increase in perplexity on language modeling tasks while reducing KV cache memory by 50%. When combined with multi-query or grouped-query attention (MQA/GQA), this approach achieves up to 96.9% KV cache compression, maintaining competitive or superior performance across both vision and language benchmarks. The method substantially lowers memory overhead, seamlessly integrates with existing head-sharing mechanisms, and offers a promising pathway for efficient on-device deployment.

0 citationsRead paper

Provenance Networks: End-to-End Exemplar-Based Explainability

Oct 02, 2025

Deep learning models suffer from poor interpretability, frequent hallucinations, and difficulty tracing predictions back to training samples—undermining their robustness and trustworthiness. To address this, we propose TraceNet, an architecture that intrinsically embeds example-based interpretability via end-to-end prediction-to-sample association. Our core innovation is a learnable weighted k-nearest neighbors (k-NN) mechanism that dynamically retrieves and weights supportive training instances in the feature space, jointly optimizing both the primary task loss and an explicit interpretability objective. TraceNet enables training-data provenance, label-noise detection, improved robustness to input perturbations, and explicit generation grounding. Experiments on medium-scale benchmarks demonstrate substantial gains in decision transparency and yield novel insights into the interplay between memorization and generalization.

0 citationsRead paper

Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions

Jan 22, 2025

State space models (SSMs) underperform in audio tasks due to limited modeling capacity, high computational overhead, and lack of inductive bias from convolutional priors. Method: We propose Centaurus—a fully SSM-based architecture that reformulates generalized SSMs as learnable tensor contraction operations with automatically optimized contraction orders; it is the first to systematically integrate group convolution, full convolution, and bottleneck structures into a heterogeneous SSM design. Contribution/Results: Centaurus is the first high-performance automatic speech recognition (ASR) model built entirely upon SSMs—without LSTM, explicit CNNs, or attention mechanisms. It outperforms homogeneous SSM baselines of comparable size on keyword spotting, speech denoising, and ASR. On ASR, it achieves state-of-the-art (SOTA) performance while significantly reducing memory consumption and computational cost during both training and inference.

0 citationsRead paper