🤖 AI Summary
Current leaderboards for large language models rely on global average performance, which inadequately captures their specific capabilities in financial services. This work proposes a meta-benchmarking framework that maps 452 publicly available benchmarks to 41 O*NET Generalized Work Activities and aggregates them into 38 BIAN banking business domains. By integrating discriminative power, coverage, and recency through a dynamic multiplicative weighting mechanism, the framework automatically downweights obsolete benchmarks. Coupled with an Elo pairwise competition algorithm, it yields normalized-free, cross-benchmark comparable scores. Validation on a June 2026 snapshot encompassing 288 models from 25 institutions demonstrates that the framework effectively supports model selection and governance decisions within financial institutions.
📝 Abstract
Public LLM leaderboards optimise for global average performance and do not capture the specific cognitive demands of financial-services work: a model that leads on MMLU-Pro may underperform on document-grounded compliance reasoning, and a coding leader may handle multi-turn customer interactions poorly. We present a meta-benchmarking framework that organises 452 publicly reported benchmarks into 41 O*NET Generalized Work Activities and aggregates those into 38 BIAN banking business domains spanning sales, operations, risk, and support work. A multiplicative weighting scheme (discrimination x coverage x recency), computed over a rolling model window, rewards benchmarks that still separate the best models, are widely reported, and remain in active use, suppressing saturated legacy tests automatically. These weights scale the K-factor in a pairwise Elo tournament, producing cross-benchmark-comparable work-activity scores without raw score normalisation; business-domain scores are weighted averages of the constituent work-activity Elos. We demonstrate the framework on a point-in-time public snapshot covering 288 models across 25 organisations as of June 2026, and describe the methodology, full taxonomy, design decisions, and limitations with the aim of making the approach reproducible for institutions facing similar selection and governance challenges.