The Backpropagation of the Wave Network

2024-11-11

AI Summary

To address the high memory consumption and low training efficiency of text understanding models, this paper proposes Wave Network—a novel token representation method based on complex-valued vectors, where magnitude encodes global semantics and phase captures the dynamic relationship between local tokens and global semantics. We first discover that waveform representations exhibit sparse and directionally biased gradient distributions during backpropagation. Leveraging this insight, we design the Token2Wave framework to jointly achieve embedding decoupling and computational efficiency. By modeling long-range dependencies via wave interference and modulation operations, and integrating memory-aware backpropagation optimization, Wave Network significantly reduces GPU memory usage and training time compared to BERT—while preserving semantic consistency (validated via [CLS] token and full-sentence gradients) and classification robustness.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

This paper provides an in-depth analysis of Wave Network, a novel token representation method derived from the Wave Network, designed to capture both global and local semantics of input text through wave-inspired complex vectors. In complex vector token representation, each token is represented with a magnitude component, capturing the global semantics of the entire input text, and a phase component, encoding the relationships between individual tokens and the global semantics. Building on prior research that demonstrated the effectiveness of wave-like operations, such as interference and modulation, during forward propagation, this study investigates the convergence behavior, backpropagation characteristics, and embedding independence within the Token2Wave framework. A detailed computational complexity analysis shows that Token2Wave can significantly reduce video memory usage and training time compared to BERT. Gradient comparisons for the [CLS] token, total input text, and classifier parameters further highlight Token2Wave's unique characteristics. This research offers new insights into wave-based token representations, demonstrating their potential to enable efficient and computationally friendly language model architectures.

Problem

• Text Processing Efficiency
• Memory Reduction
• Optimization of Text Understanding Models

Innovation

• Wave Network
• Efficiency
• Economy