FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

2026-07-13

AI Summary

This study addresses the lack of scalable, high-quality evaluation criteria for large-scale assessments of financial deep research reports, a task traditionally reliant on costly and inefficient human expert judgments. To overcome this limitation, the authors propose the first fully large language model (LLM)-driven framework for consensus-based scoring rule generation and selection. The approach automatically synthesizes candidate rules and employs a multi-LLM adjudication mechanism, augmented with consistency and discriminability filters, to distill high-confidence “gold” scoring standards. From an initial set of 14,450 candidate rules, the method identifies 2,600 effective rules that significantly differentiate performance across ten research systems, yielding pass rates ranging from 22.23% to 58.58%. Notably, the resulting rules achieve a 98.67% agreement rate with human expert judgments, substantially enhancing the scalability and automation of financial research evaluation.

Abstract

Deep research agents are increasingly used to produce long-form financial reports, yet large-scale evaluation remains bottlenecked by the need for human experts to define and execute high-quality rubrics. We address this problem by proposing a scalable pipeline for generating high-quality rubrics without human experts in the final loop. We build a financial deep research benchmark from 104 real-world user queries and automatically synthesize 14,450 query-specific candidate rubrics from model-generated reports. To justify removing human experts from rubric execution, we compare rubric judgments from three human experts with those from a three-LLM judge panel on a sampled subset, and show that LLM-based evaluation is sufficiently consistent with human evaluation to replace it for large-scale rubric screening, including 98.67\% label-level agreement on jointly unanimous items. We then derive consensus-derived gold rubrics through two filters: a strict consistency filter, which keeps a rubric only if the three LLM judges unanimously agree on every report under the same query, and a distinguishability filter, which keeps a rubric only if it assigns at least one majority-yes and at least one majority-no label across the evaluated systems. This process retains 3,687 consistency-passed rubrics, of which 2,600 remain distinguishable and form the final set of consensus-derived gold rubrics. Using this final rubric set, we obtain clearly differentiated rankings across 10 deep research systems, with item-level pass rates ranging from 58.58\% to 22.23\%. More broadly, because the pipeline removes human-expert execution from rubric generation and evaluation, it is naturally scalable for benchmark evaluation, automatic system comparison, and future studies of evaluation-driven system improvement.

Problem

• financial report evaluation
• deep research agents
• rubric generation
• human expert bottleneck
• large-scale benchmarking

Innovation

• consensus-derived rubrics
• LLM-based evaluation
• financial report benchmarking
• automated rubric generation
• deep research agents