Efficient Learning With Sine-Activated Low-rank Matrices

2024-03-28

AI Summary

To address the challenge of balancing parameter efficiency and model accuracy in low-rank decomposition neural networks, this paper proposes a novel low-rank decomposition framework incorporating sinusoidal activation functions. It is the first to introduce sine nonlinearity into the matrix decomposition process, enabling nonlinear dynamic modulation that effectively increases the representational rank without inflating parameter count. The resulting module is plug-and-play and seamlessly integrates into diverse architectures—including Vision Transformers, large language models, NeRF, and 3D shape modeling. Experiments demonstrate that, while preserving the original parameter budget, our method achieves accuracy comparable to full-rank counterparts and delivers substantial performance gains across multiple complex vision and generative tasks. This work establishes a new paradigm for designing efficient large-scale models through structured, nonlinearity-enhanced low-rank representations.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learning. These techniques significantly lower the number of parameters, striking a balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a novel theoretical framework that integrates a sinusoidal function within the low-rank decomposition process. This approach not only preserves the benefits of the parameter efficiency characteristic of low-rank methods but also increases the decomposition's rank, thereby enhancing model performance. Our method proves to be a plug in enhancement for existing low-rank models, as evidenced by its successful application in Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF) and 3D shape modelling.

Problem

• Efficient Model Design
• Low-Rank Decomposition
• Complex Task Adaptation

Innovation

• Sine-function Integrated Low-rank Decomposition
• Parameter Reduction
• Performance Enhancement