Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

2025-05-27

AI Summary

This work addresses the challenge of activation outliers in large language models (LLMs), which exacerbate quantization error, hinder model compression, and impede efficient edge deployment. We first systematically identify two root causes: the heavy-tailed nature of large-scale activation distributions and heterogeneous channel-wise sensitivity. Departing from conventional post-hoc compensation methods, we propose a lightweight, proactive suppression strategy grounded in causal analysis—integrating activation distribution modeling, channel-wise sensitivity analysis, and structured sparse reparameterization to eliminate outliers at their source. Extensive evaluation across multiple mainstream LLMs demonstrates that our approach reduces outlier count significantly, cuts INT4 quantization accuracy degradation by up to 40%, and lowers inference latency by 18%, all without additional fine-tuning. The method substantially enhances feasibility for resource-constrained edge deployment while preserving model fidelity.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Investigating outliers in large language models (LLMs) is crucial due to their significant impact on various aspects of LLM performance, including quantization and compression. Outliers often cause considerable quantization errors, leading to degraded model performance. Identifying and addressing these outliers can enhance the accuracy and efficiency of the quantization process, enabling smoother deployment on edge devices or specialized hardware. Recent studies have identified two common types of outliers in LLMs: massive activations and channel-wise outliers. While numerous quantization algorithms have been proposed to mitigate their effects and maintain satisfactory accuracy, few have thoroughly explored the root causes of these outliers in depth. In this paper, we conduct a comprehensive investigation into the formation mechanisms of these outliers and propose potential strategies to mitigate their occurrence. Ultimately, we introduce some efficient approaches to eliminate most massive activations and channel-wise outliers with minimal impact on accuracy.

Problem

• Investigating outlier impact on LLM performance and quantization
• Identifying root causes of massive activations and channel-wise outliers
• Proposing strategies to mitigate outliers with minimal accuracy loss

Innovation

• Investigates outlier formation mechanisms in LLMs
• Proposes strategies to mitigate outlier occurrences
• Introduces efficient approaches to eliminate outliers