FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting

2026-02-25

AI Summary

This study addresses the limitations of existing benchmarks for financial research reports, which fail to distinguish model capabilities across critical dimensions such as semantic consistency, data alignment, and depth of insight—hindering effective role allocation in multi-agent systems. To bridge this gap, the work proposes the first three-tier capability decomposition framework tailored for financial research, incorporating a fine-grained scoring mechanism across twelve core analytical skills that enables independent evaluation of hallucination correction and domain-specific reasoning. Through a hierarchical benchmark design, multi-criteria human evaluation, and a multi-agent capability mapping methodology, the study systematically uncovers performance disparities among closed-source, open-source general-purpose, and finance-specialized models at each capability level. The framework has been piloted in real-world financial scenarios, and all associated resources have been publicly released.

Abstract

Large language models (LLMs) are increasingly deployed in financial research workflows, where their role is evolving from single-model assistance for human analysts toward autonomous collaboration among multiple agents. Yet real-world deployments still expose factual errors, numerical inconsistencies, and shallow analysis, which can distort assessments of corporate fundamentals and trigger severe economic losses. While existing benchmarks have begun to evaluate such failures, they score all aspects of the generated analysis in one pass, failing to distinguish whether a model fails at foundational stages like auditing and correction, or underperforms at generating research-grade insights. Consequently, it obscures capability bottlenecks and the specialized strengths essential for multi-agent role assignment. To address these gaps, we introduce FinReasoning, a hierarchical benchmark that decomposes the core capabilities of financial research into semantic consistency, data alignment, and deep insight. We further propose a fine-grained evaluation framework that strengthens hallucination-correction assessment and incorporates a 12-indicator rubric for core analytical skills. FinReasoning reveals clear capability stratification across model types. Closed-source models (like Doubao-Seed-1.8) perform strongly overall and are better suited for core reasoning agents in multi-agent financial systems; open-source general models (like Qwen3-235B) show clear capability divergence and consistently underperform on Semantic Consistency, making them less suited for quality-sensitive generation tasks; financial-domain models (like Fin-R1) generate moderate insights but lack foundational auditing skills. Our work has already been deployed in pilot tests across several real-world scenarios. The resource is available at https://github.com/TongjiFinLab/FinReasoning.

Problem

• financial research
• large language models
• evaluation benchmark
• multi-agent systems
• hallucination correction

Innovation

• hierarchical benchmark
• financial reasoning
• hallucination correction
• multi-agent LLMs
• fine-grained evaluation