A Hyperbolicity Atlas of Large Language Model Hidden States

2026-09-07

AI Summary

本文通过测量10个模型的818,904个样本层数据，研究大型语言模型隐藏状态是否展示Gromov Hyperbolicity特征，揭示了层次结构随层数变化而非模型规模变化的趋势。

Abstract

LLM hidden states are ordinary vectors, but the distances among those vectors may still show hierarchical structure. To our knowledge, this paper is the first systematic study of whether prompt-token hidden states in contemporary LLMs exhibit Gromov Hyperbolicity (GH), a distance-based measure of tree-likeness. Using 818,904 sample-layer measurements from ten open-weight models across MATH500, HumanEval, WinoGrande, and TruthfulQA, we build a GH map over four axes: parameter scale, layer depth, model family, and input domain. The clearest pattern is depth, not scale: middle layers usually form a high-relative-hyperbolicity plateau, while final layers often become substantially more tree-like. Scale effects are weak and non-monotonic, matched 7/8B model families differ strongly, and domains interact with model specialization. These findings make GH useful as a practical diagnostic: it shows where hierarchical distance structure appears, how specialization changes it, and which model-layer-domain comparisons deserve closer analysis.

Problem

• Gromov Hyperbolicity
• hidden states
• large language models
• hierarchical structure

Innovation

• Gromov Hyperbolicity
• hidden states
• large language models
• hierarchical structure