BizFinBench.v2: A Unified Dual-Mode Bilingual Benchmark for Expert-Level Financial Capability Alignment

2026-01-10

arXiv.org

AI Summary

This work addresses the limitations of existing financial large language model benchmarks, which predominantly rely on synthetic or generic data and are confined to offline, static scenarios, thus failing to capture the authenticity and real-time demands of actual financial operations. To bridge this gap, we introduce FinBench, the first large-scale bilingual financial evaluation benchmark grounded in real-world stock market data from both China and the U.S. FinBench integrates offline and online evaluation modes, encompassing four core scenarios, eight foundational tasks, and two dynamic online tasks, with 29,578 expert-level question-answer pairs. Through user-query clustering, bilingual data construction, dynamic online assessment, and expert annotation, FinBench enables high-fidelity, business-grade capability disentanglement and precise alignment. Experiments reveal that while state-of-the-art models like DeepSeek-R1 perform best on online tasks, their overall accuracy—e.g., 61.5% for ChatGPT-5—remains substantially below human experts, highlighting critical deficiencies in real-world financial reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Large language models have undergone rapid evolution, emerging as a pivotal technology for intelligence in financial operations. However, existing benchmarks are often constrained by pitfalls such as reliance on simulated or general-purpose samples and a focus on singular, offline static scenarios. Consequently, they fail to align with the requirements for authenticity and real-time responsiveness in financial services, leading to a significant discrepancy between benchmark performance and actual operational efficacy. To address this, we introduce BizFinBench.v2, the first large-scale evaluation benchmark grounded in authentic business data from both Chinese and U.S. equity markets, integrating online assessment. We performed clustering analysis on authentic user queries from financial platforms, resulting in eight fundamental tasks and two online tasks across four core business scenarios, totaling 29,578 expert-level Q&A pairs. Experimental results demonstrate that ChatGPT-5 achieves a prominent 61.5% accuracy in main tasks, though a substantial gap relative to financial experts persists; in online tasks, DeepSeek-R1 outperforms all other commercial LLMs. Error analysis further identifies the specific capability deficiencies of existing models within practical financial business contexts. BizFinBench.v2 transcends the limitations of current benchmarks, achieving a business-level deconstruction of LLM financial capabilities and providing a precise basis for evaluating efficacy in the widespread deployment of LLMs within the financial domain. The data and code are available at https://github.com/HiThink-Research/BizFinBench.v2.

Problem

• financial benchmark
• large language models
• authenticity
• real-time responsiveness
• evaluation gap

Innovation

• financial benchmark
• bilingual evaluation
• online assessment
• real-world business data
• LLM capability alignment