Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling

2026-04-21

AI Summary

This work addresses the limitations of conventional Transformers in model scaling, where linear query/key/value projections constrain feature expressivity and hinder lossless retention of pre-trained knowledge. The authors propose the Nexus-Rank layer, which replaces standard attention projections with a three-stage nonlinear mapping and dual activation mechanism operating in an expanded dimensional space. Coupled with zero-initialized expansion blocks, this design enables seamless injection of new capacity along dual dimensions without disrupting existing knowledge. The approach achieves structured, lossless model expansion, with performance accurately predictable via a geometric scaling law under stable convergence trajectories. In progressive scaling from 240M to 440M parameters, the method matches Tokenformer-level perplexity on language modeling and reasoning tasks while reducing training compute by up to 41.5%.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Scaling Transformers typically necessitates training larger models from scratch, as standard architectures struggle to expand without discarding learned representations. We identify the primary bottleneck in the attention mechanism's linear projections, which strictly confine feature extraction to fixed-dimensional subspaces, limiting both expressivity and incremental capacity. To address this, we introduce Nexusformer, which replaces linear $Q/K/V$ projections with a Nexus-Rank layer, a three-stage nonlinear mapping driven by dual activations in progressively higher dimensional spaces. This design overcomes the linearity constraint and enables lossless structured growth: new capacity can be injected along two axes via zero-initialized blocks that preserve pretrained knowledge. Experiments on language modeling and reasoning benchmarks demonstrate that Nexusformer matches Tokenformer's perplexity using up to 41.5\% less training compute during progressive scaling (240M to 440M). Furthermore, our analysis of growth dynamics reveals that zero initialization induces a stable convergence trajectory, allowing us to derive a geometric scaling law that accurately predicts performance across expansion scales.

Problem

• Transformer scaling
• attention mechanism
• linear projection
• model expansion
• representation preservation

Innovation

• Nexusformer
• nonlinear attention
• structured model scaling
• zero-initialized expansion
• geometric scaling law