The Unreasonable Ineffectiveness of the Deeper Layers

2024-03-26

arXiv.org

AI Summary

This study investigates the distribution and redundancy of knowledge across deep-weight layers in large language models (LLMs). Addressing the critical question of whether deep transformer layers are necessary, we propose a block-wise layer pruning strategy grounded in inter-layer similarity analysis to systematically assess structural redundancy in mainstream open-source LLMs. We find that up to ~50% of transformer layers can be safely removed without significant performance degradation, indicating that core knowledge is predominantly concentrated in shallow layers and deep parameters are underutilized. To restore functionality efficiently, we integrate QLoRA (quantization-aware low-rank adaptation) for lightweight fine-tuning—achievable on a single A100 GPU. The pruned-and-finetuned models retain state-of-the-art performance across multiple QA benchmarks while reducing GPU memory footprint by ~50%, and significantly improving inference memory efficiency and latency. This work provides the first empirical evidence of deep-layer knowledge sparsity in LLMs and establishes a scalable, lightweight paradigm for layer compression and recovery.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed. To prune these models, we identify the optimal block of layers to prune by considering similarity across layers; then, to"heal"the damage, we perform a small amount of finetuning. In particular, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single A100 GPU. From a practical perspective, these results suggest that layer pruning methods can complement other PEFT strategies to further reduce computational resources of finetuning on the one hand, and can improve the memory and latency of inference on the other hand. From a scientific perspective, the robustness of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play a critical role in storing knowledge.

Problem

• Investigates knowledge storage in LLM weights via layer pruning.
• Explores minimal performance degradation after removing many layers.
• Uses PEFT methods for efficient experimentation on single GPU.

Innovation

• Layer pruning identifies unnecessary model parameters
• Parameter-efficient finetuning (PEFT) minimizes performance degradation
• Quantization and Low Rank Adapters (QLoRA) enable efficient experiments