What Affects the Effective Depth of Large Language Models?

2025-12-15

AI Summary

This paper identifies a pervasive “insufficient effective depth” problem in large language models (LLMs): increasing the number of layers does not proportionally enhance computational depth. Using the Qwen-2.5 series (1.5B–32B), we systematically quantify layer utilization via gradient attribution, attention entropy, and intermediate representation similarity, analyzing impacts of model scale, training paradigm (base vs. long-chain reasoning models), and task difficulty. Key findings are: (1) the effective depth ratio remains stable across scaling; (2) improved long-chain reasoning stems from extended context modeling—not deeper per-token computation; and (3) models do not dynamically allocate more layers in response to increased task difficulty. These results demonstrate the universality of ineffective depth utilization, providing empirical foundations for model pruning, adaptive early-exit mechanisms, and efficient architectural design.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The scaling of large language models (LLMs) emphasizes increasing depth, yet performance gains diminish with added layers. Prior work introduces the concept of "effective depth", arguing that deeper models fail to fully utilize their layers for meaningful computation. Building on this, we systematically study how effective depth varies with model scale, training type, and task difficulty. First, we analyze the model behavior of Qwen-2.5 family (1.5B-32B) and find that while the number of effective layers grows with model size, the effective depth ratio remains stable. Besides, comparisons between base and corresponding long-CoT models show no increase in effective depth, suggesting that improved reasoning stems from longer context rather than deeper per-token computation. Furthermore, evaluations across tasks of varying difficulty indicate that models do not dynamically use more layers for harder problems. Our results suggest that current LLMs underuse available depth across scales, training paradigms and tasks of varying difficulties, pointing out research opportunities on increasing the layer utilization rate of LLMs, model pruning, and early exiting. Our code is released at https://github.com/AheadOFpotato/what_affects_effective_depth.

Problem

• Investigates factors influencing effective depth in large language models.
• Examines how model scale, training, and task difficulty affect layer utilization.
• Identifies underutilization of depth across various models and tasks.

Innovation

• Analyzing effective depth across model scales and tasks
• Comparing base and long-CoT models for reasoning depth
• Identifying underutilized layers for pruning and early exiting