TRAWL: Tensor Reduced and Approximated Weights for Large Language Models

2024-06-25

arXiv.org

AI Summary

To address accuracy degradation and insufficient structural modeling in large language model (LLM) weight pruning, this paper proposes a training-free, cross-layer collaborative tensor decomposition method for model compression. Unlike conventional single-matrix low-rank approximation or quantization techniques, our approach is the first to extend high-order tensor decompositions—such as Tucker and CP—to jointly model weight matrices across multiple layers, enabling global structural awareness for weight denoising and compression. By enforcing collaborative low-rank approximation across layers, the method achieves simultaneous accuracy recovery and computational efficiency gains—without fine-tuning, auxiliary data, or retraining. On standard benchmarks, it improves performance by up to 16% over baseline pruning methods while substantially reducing parameter count and FLOPs. This work establishes a novel paradigm for co-optimizing accuracy and efficiency in LLM inference through structured, layer-aware tensor compression.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent research has shown that pruning large-scale language models for inference is an effective approach to improving model efficiency, significantly reducing model weights with minimal impact on performance. Interestingly, pruning can sometimes even enhance accuracy by removing noise that accumulates during training, particularly through matrix decompositions. However, recent work has primarily focused on single matrix decompositions or lower precision techniques, which may fail to fully capture structural patterns. To address these limitations, we introduce TRAWL (Tensor Reduced and Approximated Weights for Large Language Models), a technique that applies tensor decomposition across multiple weight matrices to effectively denoise LLMs by capturing global structural patterns. Our experiments show that TRAWL improves model performance by up to 16% over baseline models on benchmark datasets, without requiring additional data, training, or fine-tuning.

Problem

• Reduces large language model weights efficiently
• Enhances accuracy by removing training noise
• Captures global structural patterns via tensor decomposition

Innovation

• Tensor decomposition across multiple matrices
• Effective denoising of LLMs
• Improved model performance without additional training