Stronger Normalization-Free Transformers

2025-12-11

AI Summary

Pointwise activation functions in normalization-free Transformers suffer from inadequate design, limiting performance and generalization. Method: This paper proposes Derf(x) = erf(αx + s), a learnable pointwise nonlinearity based on the Gaussian cumulative distribution function, identified via large-scale functional space search and intrinsic property-driven analysis. Contribution/Results: Derf is the first activation to consistently outperform LayerNorm, RMSNorm, and Dynamic Tanh (DyT) across diverse multimodal tasks—including image classification/generation, speech representation learning, and DNA modeling—without normalization layers. Empirical evaluation confirms that gains stem from improved generalization, not overfitting, while maintaining structural simplicity and strong robustness to hyperparameter variation and architectural changes. Theoretically grounded and empirically validated, Derf serves as a practical, efficient, and interpretable core component for normalization-free Transformer architectures, advancing the development of explicitly normalization-free models.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Although normalization layers have long been viewed as indispensable components of deep learning architectures, the recent introduction of Dynamic Tanh (DyT) has demonstrated that alternatives are possible. The point-wise function DyT constrains extreme values for stable convergence and reaches normalization-level performance; this work seeks further for function designs that can surpass it. We first study how the intrinsic properties of point-wise functions influence training and performance. Building on these findings, we conduct a large-scale search for a more effective function design. Through this exploration, we introduce $mathrm{Derf}(x) = mathrm{erf}(alpha x + s)$, where $mathrm{erf}(x)$ is the rescaled Gaussian cumulative distribution function, and identify it as the most performant design. Derf outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains, including vision (image recognition and generation), speech representation, and DNA sequence modeling. Our findings suggest that the performance gains of Derf largely stem from its improved generalization rather than stronger fitting capacity. Its simplicity and stronger performance make Derf a practical choice for normalization-free Transformer architectures.

Problem

• Develops a point-wise function Derf to replace normalization layers in Transformers
• Explores function designs that surpass Dynamic Tanh for stable convergence and performance
• Evaluates Derf's generalization across vision, speech, and DNA modeling tasks

Innovation

• Derf function replaces normalization layers
• Large-scale search identifies erf-based activation
• Derf improves generalization across multiple domains