Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models

2024-11-06

arXiv.org

AI Summary

This work addresses the limited nonlinear expressivity and constrained dynamic modeling capability of activation functions in large language models (LLMs). We propose Polynomial Composite Activation (PolyCom), the first polynomial-based composite framework rigorously constructed within Sobolev spaces to simultaneously achieve optimal approximation rates and theoretical interpretability, enabling universal smooth function approximation with minimal parameters. By integrating approximation theory and functional analysis, we design structured polynomial activations and systematically validate them in both dense and sparse Transformer-based LLM pretraining. Experiments demonstrate that PolyCom significantly accelerates convergence and improves downstream task accuracy across diverse benchmarks, consistently outperforming mainstream baselines—including ReLU, GeLU, and SwishGLU—while preserving computational efficiency and enhancing higher-order data interaction modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Transformers have found extensive applications across various domains due to the powerful fitting capabilities. This success can be partially attributed to their inherent nonlinearity. Thus, in addition to the ReLU function employed in the original transformer architecture, researchers have explored alternative modules such as GeLU and SwishGLU to enhance nonlinearity and thereby augment representational capacity. In this paper, we propose a novel category of polynomial composition activations (PolyCom), designed to optimize the dynamics of transformers. Theoretically, we provide a comprehensive mathematical analysis of PolyCom, highlighting its enhanced expressivity and efficacy relative to other activation functions. Notably, we demonstrate that networks incorporating PolyCom achieve the $ extbf{optimal approximation rate}$, indicating that PolyCom networks require minimal parameters to approximate general smooth functions in Sobolev spaces. We conduct empirical experiments on the pre-training configurations of large language models (LLMs), including both dense and sparse architectures. By substituting conventional activation functions with PolyCom, we enable LLMs to capture higher-order interactions within the data, thus improving performance metrics in terms of accuracy and convergence rates. Extensive experimental results demonstrate the effectiveness of our method, showing substantial improvements over other activation functions. Code is available at https://github.com/BryceZhuo/PolyCom.

Problem

• Enhancing transformer dynamics with polynomial activations
• Optimizing large language models' representational capacity
• Improving approximation rates with minimal parameters

Innovation

• Polynomial Composition Activations (PolyCom)
• Optimizes transformer dynamics
• Enhances LLM performance metrics