Training-Free Activation Sparsity in Large Language Models

2024-08-26

arXiv.org

AI Summary

Existing large language models (LLMs) struggle to achieve full-model activation sparsification without costly continual pretraining, limiting inference efficiency gains. This paper introduces TEAL—a training-agnostic, magnitude-driven, full-layer activation sparsification method that requires no fine-tuning or additional training and supports mainstream architectures (e.g., Llama-2/3, Mistral) under INT4/FP8 weight quantization. TEAL jointly optimizes adaptive magnitude-based pruning with a custom sparse matrix multiplication kernel. Evaluated on models ranging from 7B to 70B parameters, it achieves 40%–50% model-level activation sparsity while incurring <1% accuracy degradation and accelerating decoding by 1.53×–1.8×. To the best of our knowledge, TEAL is the first framework enabling highly compatible, zero-training-overhead activation sparsification for LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Activation sparsity can enable practical inference speedups in large language models (LLMs) by reducing the compute and memory-movement required for matrix multiplications during the forward pass. However, existing methods face limitations that inhibit widespread adoption. Some approaches are tailored towards older models with ReLU-based sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens. This paper describes TEAL, a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up to 1.53$ imes$ and 1.8$ imes$ at 40% and 50% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.

Problem

• Activation sparsity in LLMs
• Training-free sparsity method
• Improving inference speed

Innovation

• Training-free activation sparsity method
• Magnitude-based sparsity applied globally
• Improved sparse kernels for speedups