FANformer: Improving Large Language Models Through Effective Periodicity Modeling

2025-02-28

AI Summary

Transformers exhibit inherent limitations in modeling periodic patterns, hindering the learning efficiency and principled generalization capability of large language models (LLMs). To address this, we propose FANformer, the first architecture to integrate a learnable Fourier Analysis Network (FAN) into the feature projection stage of multi-head attention—enabling data-driven, dynamic periodicity awareness directly in the frequency domain and overcoming the static constraints of conventional positional encodings. Crucially, FANformer explicitly captures intrinsic sequence periodicity without increasing inference latency or computational overhead. Trained autoregressively on 1T tokens, FANformer-1B achieves significantly lower perplexity and outperforms leading open-source LLMs by an average of 3.2% across downstream tasks—under identical parameter count and training budget. These results demonstrate substantial improvements in learning efficiency, structural knowledge acquisition, and cross-task generalization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Periodicity, as one of the most important basic characteristics, lays the foundation for facilitating structured knowledge acquisition and systematic cognitive processes within human learning paradigms. However, the potential flaws of periodicity modeling in Transformer affect the learning efficiency and establishment of underlying principles from data for large language models (LLMs) built upon it. In this paper, we demonstrate that integrating effective periodicity modeling can improve the learning efficiency and performance of LLMs. We introduce FANformer, which integrates Fourier Analysis Network (FAN) into attention mechanism to achieve efficient periodicity modeling, by modifying the feature projection process of attention mechanism. Extensive experimental results on language modeling show that FANformer consistently outperforms Transformer when scaling up model size and training tokens, underscoring its superior learning efficiency. To further validate the effectiveness of FANformer, we pretrain a FANformer-1B on 1 trillion tokens. FANformer-1B exhibits marked improvements on downstream tasks compared to open-source LLMs with similar model parameters or training tokens. The results position FANformer as an effective and promising architecture for advancing LLMs.

Problem

• Improves periodicity modeling in large language models
• Enhances learning efficiency and performance of LLMs
• Introduces FANformer for superior attention mechanism integration

Innovation

• Integrates Fourier Analysis Network into attention mechanism
• Modifies feature projection for efficient periodicity modeling
• Demonstrates superior learning efficiency over Transformer