InvestPhilBench: A Multi-Layer Dynamic Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy

2026-06-24

AI Summary

This study addresses the lack of effective benchmarks for evaluating large language models’ ability to reconstruct and apply expert investors’ procedural decision-making frameworks. The authors propose the first dynamic evaluation benchmark encompassing eight cognitive levels—from principle identification to out-of-framework extrapolation—grounded in real-world investment philosophies to systematically assess procedural reasoning capabilities. They introduce Gate-level Reasoning Accuracy (GRA) to uncover structural deficiencies masked by aggregate scores and develop an automated scoring pipeline integrating five algorithmic metrics, a Failure Mode Detection Protocol (FMDP), and model-in-the-loop retrieval. Experiments reveal pronounced performance stratification: while overall scores appear saturated, GRA exposes significant deficits in higher-level reasoning (L4 GRA ≈ 0.77; L7 GRA ≈ 0.57–0.62). The automated scoring shows strong agreement with human judgments (r = 0.72).

Abstract

Large language models are increasingly deployed as investment research assistants, yet no benchmark tests whether they can accurately reconstruct and apply the specific procedural decision frameworks of expert investors. We introduce InvestPhilBench, a multi-layer dynamic benchmark spanning eight cognitive tiers, from principle identification (L1) to novel framework extrapolation (L8). The v0.6 release comprises 118 primary-source-verified investment principle cards, 25 decision framework cards with explicit topology metadata, and 243 QA questions (197 dev / 46 held-out test). For reproducible scoring at scale we introduce the Benchmark Automated Scoring Pipeline (BASP) -- five algorithmic metrics (OGRS, KCCS, SAP@k, IVP, CKCA) -- the Failure Mode Detection Protocol (FMDP) with computable rules for six failure modes, and Gate Reconstruction Accuracy (GRA), a per-gate metric for questions with gold reasoning programs. In this release, InvestPhilBench is primarily a benchmark-and-methodology contribution. A four-model sanity wave on the 188-question development split shows a sharp provider-tier split (BASP 0.906 vs. 0.438); these mixed-judge numbers are confounded upper bounds. The central finding: the BASP composite saturates at the frontier (Claude L4 = 0.932) while GRA still exposes a procedural deficit (frontier L4 GRA approx. 0.77, L7 GRA 0.57-0.62) -- composite scoring rewards fluent prose and hides the procedural gap. v0.6 implements a unified judge and true model-in-the-loop retrieval/oracle conditions; the de-confounded multi-model leaderboard and full three-condition run are v1.0 deliverables. On a 100-item expert-annotated gold set the automated BASP composite tracks the human reference at Pearson r = 0.72 (MAE = 0.10), with attribution (SAP@3) the weakest sub-metric and the failure-mode detector running sensitive-but-over-flagging.

Problem

• procedural reasoning
• investment philosophy
• large language models
• benchmark
• decision frameworks

Innovation

• procedural reasoning
• dynamic benchmark
• automated scoring pipeline
• investment philosophy
• failure mode detection