Problem
Research questions and friction points this paper is trying to address.
financial intelligence
AI agent benchmarking
investment workflow
open-ended evaluation
frontier models
FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents
🤖 AI Summary
Existing benchmarks for financial agents are limited to isolated stages of the investment process and fail to evaluate complex reasoning capabilities in realistic settings. This work proposes the first open-ended evaluation framework encompassing the entire investment workflow, featuring 220 expert-crafted multidimensional queries and 11,543 traceable scoring rules across six core use cases. The framework introduces standardized tool-call interfaces grounded in public data and employs an LLM-as-a-judge mechanism calibrated with human oversight. Experiments reveal that toolchain design significantly impacts performance: the Samaya system achieves a leading score of 56.0%, while the open-source model Kimi K3 reaches 46.4%—comparable to top proprietary models at 4.5× lower cost. However, screening and macroeconomic analysis remain key bottlenecks, with even the best system scoring only 33% and 39%, respectively. The dataset and scoring code are publicly released.
📝 Abstract
AI agents are increasingly deployed for professional investment research, yet no benchmark captures the complexity of the full investor workflow. Existing benchmarks mainly target financial data extraction, a narrow slice that current models have largely saturated, while reference-based metrics and generic LLM-as-a-judge scoring fall short on the open-ended, long-form answers that real analyst queries demand. We introduce FrontierFinance, a fully open benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow. FrontierFinance is both broader and harder than existing public finance benchmarks. Evaluating frontier models and agent systems under a common harness restricted to publicly available data, we find that the tool harness, not the model alone, strongly shapes quality and efficiency; that Samaya's in-house system leads at 56.0%, ahead of the strongest frontier model (Claude Fable 5, 49.2%) at roughly 2.2x lower cost; and that the best open-weight model (Kimi K3, 46.4%) nearly matches the best proprietary model at 4.5x lower cost. Screening & Discovery and Sector, Industry & Macro remain the hardest use cases across all systems, where even the best systems reach only 33% and 39%. We make the dataset and grading code publicly available.
Innovation
Methods, ideas, or system contributions that make the work stand out.
FrontierFinance
financial agent benchmark
open-ended evaluation
tool-augmented reasoning
investment workflow
🔎 Similar Papers
No similar papers found.
