ZetaGPT: A Reference Implementation of Positional--Encoding--Free State--Space--Attention Language Models

2026-08-10

AI Summary

Traditional Transformers rely on explicit positional encodings to capture word order, lacking intrinsic positional awareness. This work proposes a hybrid architecture that integrates a causal state space model before the self-attention layers, enabling implicit encoding of sequential positional information through recurrent state dynamics. Consequently, the model effectively processes ordered text without any explicit positional encodings. We introduce ZetaGPT, the first open-source small language model entirely free of positional encodings, accompanied by an end-to-end reproducible framework encompassing tokenizer training, pretraining, supervised fine-tuning, RLHF, and pure reinforcement learning–based chain-of-thought reasoning. This framework facilitates efficient prototyping and algorithmic validation while significantly simplifying model design without compromising representational capacity.

Abstract

Transformer-based language models rely on self-attention, whose computation is permutation-equivariant and therefore lacks an intrinsic mechanism for representing token order. Existing architectures address this limitation by explicitly incorporating positional information through learned positional embeddings or hand-crafted positional encodings, such as rotary positional encoding (RoPE), treating positional information as an architecturally acquired capability rather than an inherent property of the model. Motivated by the pursuit of positional-encoding-free architectures, this work explores a language model architecture that integrates causal state-space equations to implicitly encode positional information before attention computation. Specifically, each model block applies a causal state-space equation before self-attention, allowing recurrent state dynamics to encode sequential information into token representations. Consequently, subsequent attention layers operate on position-aware representations without requiring explicit positional encodings while retaining the expressive modeling capacity of self-attention. We present \textsc{ZetaGPT}, a compact hybrid language model designed for research, rapid prototyping, algorithm verification, and educational applications. In addition to the proposed architecture, \textsc{ZetaGPT} provides a fully open-source, end-to-end training pipeline encompassing dataset construction, tokenizer training, pretraining, supervised fine-tuning, reinforcement learning from human feedback (RLHF), and chain-of-thought (CoT) reasoning via pure reinforcement learning. To the best of our knowledge, \textsc{ZetaGPT} is the first open-source small language model without explicit positional encoding and establishes a compact, reproducible reference implementation for the development and empirical study of positional-encoding-free language models.

Problem

• positional encoding
• language models
• self-attention
• sequence order
• state-space models

Innovation

• positional-encoding-free
• state-space models
• causal state-space equations
• self-attention
• language modeling