perception

This work clarifies the functional boundaries and necessity of core Transformer components—tokenization, embedding/un-embedding, masking, positional encoding, and padding—addressing widespread conceptual ambiguity in their mechanistic roles. Targeting ML engineers, we propose an incremental, invertibility-based analytical framework: using binary (0/1) sequences as probes, we systematically introduce each component via a “zero-one construction” and empirically validate its irreplaceability in the encode-decode pipeline. Implemented lightweightly in PyTorch, our framework supports manual attention matrix construction, explicit positional embedding injection, and interpretable mask design. Experiments demonstrate significantly improved conceptual accuracy among learners. Notably, we provide the first empirical verification that, in the absence of self-attention, positional encoding combined with padding alone suffices for basic length-aware tasks.

Standard Transformer self-attention relies on dot-product similarity, which lacks expressivity for capturing complex, nonlinear token interactions. To address this, we propose Neural Attention—a novel attention mechanism that replaces the fixed dot-product with a learnable feed-forward neural network to compute attention weights. This is the first approach to introduce parameterized nonlinear mapping into attention weight computation while preserving dimensional compatibility and mathematical differentiability, thereby substantially enhancing representational capacity. The mechanism is inherently cross-modal: it achieves consistent gains across NLP and CV benchmarks—reducing perplexity by over 5% on WikiText-103 and significantly improving classification accuracy on CIFAR-10/100. Through rigorous computational complexity analysis and architectural optimization, we ensure both enhanced modeling capability and practical deployability.

This work addresses the limited expressiveness of linear self-attention in in-context learning. We propose an extended linear self-attention mechanism augmented with a learnable bias matrix, overcoming the fundamental limitation of standard linear attention—which can only approximate matrix multiplication—while preserving O(n) time complexity. For the first time, our mechanism exactly represents key linear operators, including precise matrix multiplication and identity mapping (i.e., skip connections). Through formal matrix-theoretic analysis and heuristic input-structure design, we prove its capacity to represent constant matrices, input matrices, and arbitrary two- or three-matrix products. Furthermore, we construct an interpretable, batched ridge regression gradient-descent module grounded in this mechanism; empirical evaluation on few-shot regression tasks demonstrates algorithm-level generalization. This establishes a foundation for native, interpretable algorithmic reasoning within Transformer architectures.

This work addresses the significant performance gap of small-scale Transformers—compared to LSTMs and MLPs—under task-switching scenarios (e.g., dynamically triggered incremental, additive, reverse-copy, and context operations via control tokens). To this end, we propose Cisformer, a novel architecture incorporating non-translation-invariant positional encodings to better capture control-token semantics, and Extensive Attention, which expands the attention scope and explicitly models task-switching logic. Evaluated on our newly constructed, bounded-domain arithmetic benchmark IARC, the Cisformer–Extensive Attention combination achieves ≈95% accuracy—substantially surpassing standard Transformers (<50%), LSTMs (≈70%), and MLPs (≈70%). Our findings demonstrate that embedding structured inductive biases into attention mechanisms can effectively overcome the performance bottlenecks of lightweight sequence models in dynamic task-switching settings, establishing a new paradigm for task-adaptive design of resource-efficient architectures.

This work investigates the specific functional contributions of self-attention in Transformers to algorithmic tasks—namely mathematical reasoning, memory, and retrieval—and clarifies its division of labor with MLPs. To isolate learnable mechanisms, the authors systematically freeze key components (Q/K projectors, MLPs) and introduce MixiT, a variant with fixed random attention weights. Key findings are: (1) MixiT matches fully trained models on basic arithmetic and strong-memory tasks, demonstrating for the first time that random attention suffices for these capabilities; (2) freezing Q/K projectors preserves induced attention heads and competitive language modeling performance, indicating robustness to input projection learning; (3) input-dependent attention is indispensable for retrieval tasks. Collectively, results reveal that self-attention’s core role lies in structured information routing—not merely in learned weight optimization—highlighting its architectural, rather than purely parametric, utility.

This study investigates the performance discrepancy between linear and quadratic attention mechanisms in in-context learning (ICL) for linear regression tasks. Building upon the standard Transformer architecture, we conduct a systematic comparison under controlled conditions to evaluate their relative efficacy in learning quality, convergence behavior, and generalization capability, while also examining the influence of model depth on ICL performance. Using mean squared error (MSE) as the primary evaluation metric, our experiments reveal that although both attention mechanisms exhibit comparable overall performance in ICL, linear attention encounters specific performance bottlenecks under certain configurations. Furthermore, model depth is found to significantly modulate ICL effectiveness. These findings elucidate both the advantages and limitations of linear attention in ICL settings, offering empirical insights to guide the design of efficient attention mechanisms.

This work addresses the high computational cost of Transformers in processing long sequences, which stems from the linear growth of fast weight memory in attention mechanisms with sequence length. The authors reformulate attention as a dynamically instantiated two-layer fast-weight MLP and introduce the MiTA mechanism: by compressing the original wide MLP through a small set of landmark queries and integrating Top-k activated key-value pairs to construct deformable experts, MiTA unifies model compression with sparse routing. This approach represents the first effort to embed efficient attention within a fast-weight scaling framework, significantly reducing computational complexity while preserving expressive capacity. Preliminary vision experiments demonstrate MiTA’s effectiveness and potential in long-context scenarios.

This work proposes a novel perspective on sequence modeling by reinterpreting the attention mechanism as a dynamic parameter prediction process within a multilayer perceptron (MLP). Traditional Transformers rely on explicit attention for global context modeling, yet their quadratic computational complexity hinders scalability. The proposed approach entirely eliminates explicit attention, instead implicitly compressing global contextual information through dynamically generated MLP parameters, thereby achieving linear computational complexity. This is the first method to fully replace the attention mechanism with a dynamic parameterization strategy while preserving strong global modeling capabilities. Empirical results demonstrate that the model significantly reduces computational overhead in vision tasks without sacrificing performance, matching or closely approaching that of standard Transformers.

This study addresses the computational and interpretability limitations of Transformers by systematically reviewing and categorizing forty non-attention-based vision models, including architectures based on convolutional networks, multilayer perceptrons (MLPs), and state space models. It establishes, for the first time, a unified taxonomy for non-Transformer visual approaches and provides a comprehensive evaluation across four key dimensions: efficiency, scalability, interpretability, and robustness. The analysis elucidates the respective strengths and weaknesses of each model class, clarifies their competitive standing across multiple performance metrics, and identifies both current challenges and promising directions for future research. This work thus offers a foundational framework and strategic guidance for advancing efficient and interpretable vision models.