Spectral Signatures of Large Language Models

2026-07-03

AI Summary

This work addresses the lack of unified, scalable quantitative methods for lineage tracing, license management, and performance evaluation of large language models (LLMs) by proposing the first model-level signature based on spectral shape. Leveraging heavy-tailed self-regularization theory, the method extracts a compact, data-agnostic, and scale-invariant spectral signature from the empirical spectral density of model weights, exhibiting robustness in post-training settings. Through empirical spectral density analysis, heavy-tailed distribution modeling, and spectral shape metrics, experiments on a large-scale open-source LLM corpus demonstrate that the proposed signature effectively enables unsupervised clustering, lineage attribution, and performance trend prediction, thereby validating its practical utility and effectiveness for managing large models.

Abstract

The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation. However, task-specific benchmarks are insufficient for this setting, as LLMs differ widely in architectures, scales, and training procedures. To address this challenge, we adopt spectral shape-based metrics for managing and quantifying LLMs based on Heavy-Tailed Self-Regularization theory. Our approach uses the shape information of the weight empirical spectral density as a compact spectral signature of each model. This signature captures intrinsic properties of pretrained models and remains robust during post-training, making it suitable for model-level analysis. In addition, this metric is data-free, computationally-efficient, and scale-invariant, enabling large-scale analysis in practice. Moreover, we curate a large and diverse model corpus consisting of major open-source LLM families, and use it to systematically benchmark spectral and non-spectral metrics across models and downstream tasks. We show that our spectral signature supports the tracking of the model lineage, the unsupervised clustering of similar models, and the quantification of the model performance. Overall, the proposed spectral signature provides a meaningful proxy for broad performance trends across LLMs, enabling efficient organization, comparison, and analysis of large model collections.

Problem

• large language models
• model management
• spectral signatures
• model lineage
• model quantification

Innovation

• spectral signature
• Heavy-Tailed Self-Regularization
• empirical spectral density
• model lineage tracing
• scale-invariant metric