Selected work

Representative Papers

Avey-B

Feb 17, 2026

This work addresses the challenge of constructing efficient bidirectional encoders for resource-constrained industrial settings by reconfiguring the attention-free Avey model into a pure encoder architecture. It introduces three key innovations: decoupling static and dynamic parameters, a stability-oriented normalization strategy, and neural compression techniques. The proposed approach achieves high-performance bidirectional contextual modeling without attention mechanisms for the first time, consistently outperforming four mainstream Transformer-based encoders on standard token classification and information retrieval benchmarks. Furthermore, it demonstrates superior scaling efficiency and computational efficacy in long-context tasks, offering a compelling alternative to conventional attention-based architectures in scenarios where computational resources are limited.

0 citationsRead paper

Don't Pay Attention

Jun 12, 2025

Transformers face fundamental limitations in modeling ultra-long sequences due to fixed context windows and quadratic computational complexity; RNNs, while linear in sequence length, suffer from poor parallelizability. This paper introduces Avey, the first architecture to entirely abandon both attention and recurrence paradigms. Instead, it synergistically combines a token-rank sorter with a lightweight autoregressive neural processor to perform position-agnostic critical token selection and modeling—thereby fully decoupling context width from sequence length. This design enables unbounded-length long-range dependency modeling with near-linear time complexity. Experiments demonstrate that Avey matches Transformer performance on standard short-context NLP tasks, while substantially outperforming state-of-the-art models on long-range benchmarks—including PG19 and the Long Range Arena—validating both the efficacy and scalability of this novel architectural paradigm.

0 citationsRead paper

Recent publications

Latest Papers

Avey-B

Feb 17, 2026

This work addresses the challenge of constructing efficient bidirectional encoders for resource-constrained industrial settings by reconfiguring the attention-free Avey model into a pure encoder architecture. It introduces three key innovations: decoupling static and dynamic parameters, a stability-oriented normalization strategy, and neural compression techniques. The proposed approach achieves high-performance bidirectional contextual modeling without attention mechanisms for the first time, consistently outperforming four mainstream Transformer-based encoders on standard token classification and information retrieval benchmarks. Furthermore, it demonstrates superior scaling efficiency and computational efficacy in long-context tasks, offering a compelling alternative to conventional attention-based architectures in scenarios where computational resources are limited.

0 citationsRead paper

Don't Pay Attention

Jun 12, 2025

Transformers face fundamental limitations in modeling ultra-long sequences due to fixed context windows and quadratic computational complexity; RNNs, while linear in sequence length, suffer from poor parallelizability. This paper introduces Avey, the first architecture to entirely abandon both attention and recurrence paradigms. Instead, it synergistically combines a token-rank sorter with a lightweight autoregressive neural processor to perform position-agnostic critical token selection and modeling—thereby fully decoupling context width from sequence length. This design enables unbounded-length long-range dependency modeling with near-linear time complexity. Experiments demonstrate that Avey matches Transformer performance on standard short-context NLP tasks, while substantially outperforming state-of-the-art models on long-range benchmarks—including PG19 and the Long Range Arena—validating both the efficacy and scalability of this novel architectural paradigm.

0 citationsRead paper