Mapping and Measuring the Behavioral Evolution of Large Language Models

2026-08-11

AI Summary

Existing benchmarks primarily assess model performance but struggle to uncover behavioral relationships and evolutionary patterns among large language models. This work proposes a label-free, sentence-level framework for measuring behavioral divergence based on responses to 10,000 shared prompts. The approach integrates embeddings, principal component analysis, Gromov–Wasserstein distance, maximum mean discrepancy, and hierarchical clustering, with geometric order preservation validated under multi-encoder compression. For the first time, it unifies diverse alignment and invariance assumptions through a measure-theoretic lens, establishing architecture-agnostic theoretical conditions on the training side to explain behavioral similarity. Key findings reveal that model families form distinct clusters (with gpt-2 as a global outlier), cross-family behavioral distances converge over time, recent reasoning models exhibit more compact response distributions, and token-level and sentence-level results show strong agreement (Spearman ρ = 0.98).

Abstract

Benchmark leaderboards summarize how well a language model performs, but not how its behavior relates to that of other models or changes across generations. We characterize the output behavior of 32 models from six families using their responses to a shared bank of 10{,}000 prompts. After embedding each response, we construct three complementary sentence-level dissimilarities: an aligned mean per-prompt distance, which is a pseudometric on observed model responses; a PCA-compressed summary of prompt-wise disagreement; and an alignment-free Gromov--Wasserstein discrepancy between models' internal response geometries. We use these constructions to study static organization and temporal change on a release-date axis through behavioral maps, family-wise drift, hierarchical clustering, cross-family convergence, and response-cloud dispersion. Across the three constructions, model families form coherent clusters, with \texttt{gpt-2} as a global outlier; cross-family distances decrease over time; and several recent reasoning-oriented models have comparatively compact response clouds. A token-level cross-check based on per-prompt Maximum Mean Discrepancy closely agrees with the sentence-level mean distance (Spearman $ρ=0.98$) and recovers the same qualitative findings. We organize these comparisons through a measure-theoretic lens making their alignment and invariance assumptions explicit. We also establish an architecture-agnostic sufficient condition linking behavioral similarity to inference-prompt coverage, small excess population log-loss, and similar effective target distributions---a possible training-side account rather than an empirical explanation of the observed trends. Our pipeline is label-free, and re-encoding every response with three further encoders---down to one $73\times$ smaller---preserves the rank geometry, the outliers, and the sign of the time trend.

Problem

• behavioral evolution
• large language models
• model comparison
• temporal change
• response geometry

Innovation

• behavioral evolution
• Gromov–Wasserstein discrepancy
• label-free evaluation
• measure-theoretic framework
• model clustering