XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning

2025-08-20

AI Summary

This paper addresses the limitations of large language models (LLMs) in graduate-level, knowledge-intensive financial tasks—including multimodal reasoning, time-series forecasting, scenario planning, and numerical modeling—by introducing XFinBench, the first multimodal evaluation benchmark for advanced finance (4,235 samples), accompanied by a domain-specific knowledge base covering 3,032 financial terms. Methodologically, it integrates textual, tabular, and visual inputs within a knowledge-enhanced framework coupled with fine-grained error analysis. Key contributions are: (1) a five-dimensional capability evaluation taxonomy; (2) empirical evidence that knowledge injection yields greater performance gains for smaller models; and (3) identification of computational precision and chart interpretation as critical bottlenecks. Experiments show the best pure-text model (o1) achieves only 67.3% overall accuracy—12.5 percentage points below human experts—highlighting fundamental gaps in LLMs’ capacity for complex financial reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Solving financial problems demands complex reasoning, multimodal data processing, and a broad technical understanding, presenting unique challenges for current large language models (LLMs). We introduce XFinBench, a novel benchmark with 4,235 examples designed to evaluate LLM's ability in solving complex, knowledge-intensive financial problems across diverse graduate-level finance topics with multi-modal context. We identify five core capabilities of LLMs using XFinBench, i.e, terminology understanding, temporal reasoning, future forecasting, scenario planning, and numerical modelling. Upon XFinBench, we conduct extensive experiments on 18 leading models. The result shows that o1 is the best-performing text-only model with an overall accuracy of 67.3%, but still lags significantly behind human experts with 12.5%, especially in temporal reasoning and scenario planning capabilities. We further construct a knowledge bank with 3,032 finance terms for knowledge augmentation analysis, and find that relevant knowledge to the question only brings consistent accuracy improvements to small open-source model. Additionally, our error analysis reveals that rounding errors during calculation and blindness to position and intersection of curves in the image are two primary issues leading to model's poor performance in calculating and visual-context questions, respectively. Code and dataset are accessible via GitHub: https://github.com/Zhihan72/XFinBench.

Problem

• Benchmarking LLMs on complex financial problem-solving tasks
• Evaluating multimodal data processing in financial reasoning
• Assessing knowledge-intensive capabilities across graduate finance topics

Innovation

• Novel benchmark for LLM financial problem evaluation
• Knowledge bank integration for financial terminology augmentation
• Multimodal analysis combining text and visual data processing