Selected work

Representative Papers

Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing

May 29, 2026

This work addresses the challenge in autoregressive generation of simultaneously achieving fast decoding, low memory overhead, and effective long-range dependency modeling. To this end, the authors propose a unified structured generalized linear recurrence framework that decouples the direct single-step input–output influence from the multi-step state propagation mechanism. By incorporating recurrence designs that depend on multiple historical states, the framework substantially enhances model expressivity while maintaining controllable computational complexity. This formulation generalizes both state space models and attention mechanisms, establishing a unified token mixing paradigm. Empirical validation on synthetic tasks and language modeling benchmarks demonstrates the framework’s efficiency and strong representational capacity, offering a novel toolkit for designing high-performance token mixers.

0 citationsRead paper

Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity

May 21, 2026

This work addresses the challenges of efficiently maintaining entity states over long sequences and the high computational cost of dense attention mechanisms. To this end, the authors propose a block-sparse attention scheme that exploits the block-diagonal dominant structure inherent in attention distributions for entity tracking tasks. By introducing a resolvent-based block approximation method, the approach enables multi-hop state propagation within a single layer. Notably, it achieves sub-quadratic sequence complexity—specifically O(n^{4/3}d)—while preserving accuracy on par with dense attention. Experimental results demonstrate that the method reduces runtime by 12–29% compared to standard dense Transformers, with peak speedups reaching 2.4×.

0 citationsRead paper

Forward Only Learning for Orthogonal Neural Networks of any Depth

Dec 19, 2025

Traditional backpropagation (BP) suffers from high computational overhead and poor scalability to deep networks; existing forward-only methods (e.g., PEPITA) are constrained to shallow architectures and cannot support deep training. This paper proposes FOTON, the first framework to theoretically identify the root cause of forward-training failure and establish a linear orthogonality condition equivalent to BP—relaxed nonlinearly to enable depth scalability. FOTON integrates orthogonal weight constraints, forward-gradient approximation, implicit differentiation, and convolution-aware structural design. It significantly outperforms PEPITA on both fully connected and convolutional networks, successfully training models exceeding 100 layers while achieving substantial speedup in training time. The implementation is publicly available.

0 citationsRead paper

Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics

Jul 03, 2025

Standard Transformers struggle with high-resolution inputs due to the quadratic complexity of self-attention, and existing patching or downsampling strategies often compromise fine-grained details. To address this, we propose the Multipole Attention Neural Operator (MANO), which models attention as distance-based, multi-scale interactions among grid points—introducing the fast multipole method from *n*-body simulations into Transformer architecture for the first time. MANO preserves a global receptive field per attention head while achieving linear time and memory complexity, eliminating the need for downsampling and retaining the finest structural details. By integrating structural priors from both vision and physical simulation, it balances local precision with global modeling. On image classification and Darcy flow simulation benchmarks, MANO matches the accuracy of ViT and Swin Transformer while reducing runtime and peak memory consumption by one to two orders of magnitude.

0 citationsRead paper

Fast Training of Recurrent Neural Networks with Stationary State Feedbacks

Mar 29, 2025

Recurrent Neural Networks (RNNs) suffer from high computational overhead during training due to backpropagation through time (BPTT), limiting scalability. This work proposes SS-RNN, a novel RNN architecture that eliminates BPTT by integrating principles from state space models (SSMs). Leveraging the assumption of temporal stationarity, SS-RNN employs a static, structured state feedback matrix, reformulating gradient propagation as a forward-mode linear transformation. This constitutes the first incorporation of SSM mechanisms into the RNN gradient computation paradigm. Crucially, SS-RNN preserves effective long-range dependency modeling while enabling efficient, approximate gradient estimation. Empirically, on standard language modeling benchmarks, SS-RNN achieves perplexity competitive with Transformers at comparable parameter counts, while reducing training cost significantly and accelerating inference.

0 citationsRead paper

Recent publications

Latest Papers

Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing

May 29, 2026

This work addresses the challenge in autoregressive generation of simultaneously achieving fast decoding, low memory overhead, and effective long-range dependency modeling. To this end, the authors propose a unified structured generalized linear recurrence framework that decouples the direct single-step input–output influence from the multi-step state propagation mechanism. By incorporating recurrence designs that depend on multiple historical states, the framework substantially enhances model expressivity while maintaining controllable computational complexity. This formulation generalizes both state space models and attention mechanisms, establishing a unified token mixing paradigm. Empirical validation on synthetic tasks and language modeling benchmarks demonstrates the framework’s efficiency and strong representational capacity, offering a novel toolkit for designing high-performance token mixers.

0 citationsRead paper

Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity

May 21, 2026

This work addresses the challenges of efficiently maintaining entity states over long sequences and the high computational cost of dense attention mechanisms. To this end, the authors propose a block-sparse attention scheme that exploits the block-diagonal dominant structure inherent in attention distributions for entity tracking tasks. By introducing a resolvent-based block approximation method, the approach enables multi-hop state propagation within a single layer. Notably, it achieves sub-quadratic sequence complexity—specifically O(n^{4/3}d)—while preserving accuracy on par with dense attention. Experimental results demonstrate that the method reduces runtime by 12–29% compared to standard dense Transformers, with peak speedups reaching 2.4×.

0 citationsRead paper

Forward Only Learning for Orthogonal Neural Networks of any Depth

Dec 19, 2025

Traditional backpropagation (BP) suffers from high computational overhead and poor scalability to deep networks; existing forward-only methods (e.g., PEPITA) are constrained to shallow architectures and cannot support deep training. This paper proposes FOTON, the first framework to theoretically identify the root cause of forward-training failure and establish a linear orthogonality condition equivalent to BP—relaxed nonlinearly to enable depth scalability. FOTON integrates orthogonal weight constraints, forward-gradient approximation, implicit differentiation, and convolution-aware structural design. It significantly outperforms PEPITA on both fully connected and convolutional networks, successfully training models exceeding 100 layers while achieving substantial speedup in training time. The implementation is publicly available.

0 citationsRead paper

Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics

Jul 03, 2025

Standard Transformers struggle with high-resolution inputs due to the quadratic complexity of self-attention, and existing patching or downsampling strategies often compromise fine-grained details. To address this, we propose the Multipole Attention Neural Operator (MANO), which models attention as distance-based, multi-scale interactions among grid points—introducing the fast multipole method from *n*-body simulations into Transformer architecture for the first time. MANO preserves a global receptive field per attention head while achieving linear time and memory complexity, eliminating the need for downsampling and retaining the finest structural details. By integrating structural priors from both vision and physical simulation, it balances local precision with global modeling. On image classification and Darcy flow simulation benchmarks, MANO matches the accuracy of ViT and Swin Transformer while reducing runtime and peak memory consumption by one to two orders of magnitude.

0 citationsRead paper

Fast Training of Recurrent Neural Networks with Stationary State Feedbacks

Mar 29, 2025

Recurrent Neural Networks (RNNs) suffer from high computational overhead during training due to backpropagation through time (BPTT), limiting scalability. This work proposes SS-RNN, a novel RNN architecture that eliminates BPTT by integrating principles from state space models (SSMs). Leveraging the assumption of temporal stationarity, SS-RNN employs a static, structured state feedback matrix, reformulating gradient propagation as a forward-mode linear transformation. This constitutes the first incorporation of SSM mechanisms into the RNN gradient computation paradigm. Crucially, SS-RNN preserves effective long-range dependency modeling while enabling efficient, approximate gradient estimation. Empirically, on standard language modeling benchmarks, SS-RNN achieves perplexity competitive with Transformers at comparable parameter counts, while reducing training cost significantly and accelerating inference.

0 citationsRead paper