Mapping Overlaps in Benchmarks through Perplexity in the Wild

2025-09-27

AI Summary

This work addresses the challenge of quantifying capability overlap across LLM benchmarks. We propose *Capability Familiarity Signatures*—a benchmark representation method grounded in in-the-wild perplexity—designed to disentangle underlying cognitive capabilities. Leveraging performance of 32 large language models on real-world corpora, we apply sequential forward selection and linear regression to identify highly predictive, statistically significant tokens from 88 diverse benchmarks, effectively isolating capability signals from task formatting and content biases. Results reveal substantial interdependence among logical, mathematical, and linguistic abilities, while coding ability exhibits the highest modularity; multilingual/cultural benchmarks show strong internal heterogeneity; and conventional correlation-based analyses are vulnerable to non-capability confounders. To our knowledge, this is the first framework enabling interpretable, computationally tractable modeling of cross-benchmark capability overlap—establishing a new paradigm for LLM capability evaluation and benchmark design.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We develop signatures of capacity familiarity to characterize large language model (LLM) benchmarks and their meaningful overlaps. Benchmark signatures probe the capacity required for benchmark performance. We formally define them as a set of salient tokens drawn from in-the-wild, naturally authored corpora, where LLM token perplexity, reflecting more or less pre-training exposure, becomes highly predictive of LLM benchmark performance. Through a large-scale meta-evaluation, we extract benchmark signatures via stepwise forward selection with linear regressions across 32 LLMs and 88 benchmarks spanning diverse knowledge, coding, logic, instruction following, math, language, reasoning, and world modeling. Our analysis situates signatures in relation to both the semantic similarity of benchmark questions and the correlation of model performance. While performance overlaps are universally high and semantic overlaps remain confined to a narrow mid-range, benchmark signatures prove highly informative in capturing variation, overlap, and divergence. We observe overlap in knowledge and reasoning subtasks, whereas multilingual and cultural benchmarks exhibit less similarity, even compared to cross-task overlap. Notably, performance-level results are strongly influenced by benchmark-orthogonal factors such as question format, highlighting limitations in LLM generalization, the conflation of performance with ability, and issues inherent in current mainstream benchmark agreement studies. Benchmark signatures, however, remain robust to such effects. Ultimately, we identify cross-functional overlaps across logic, math, language, instruction following, and world modeling, with coding emerging as the least overlapping domain. Together, these findings provide mechanistic insights into benchmark validity and LLM sensitivities, and sketch the underlying landscape of interconnected LLM capabilities.

Problem

• Characterizing meaningful overlaps in LLM benchmarks using capacity familiarity signatures
• Identifying cross-functional overlaps across logic, math, language, and world modeling capabilities
• Addressing limitations in LLM generalization and benchmark agreement studies

Innovation

• Benchmark signatures use token perplexity from corpora
• Extract signatures via stepwise selection across models
• Signatures capture capability overlaps while resisting format bias