V-FiLLM: Verified Financial LLM Reasoning Benchmark

2026-08-11

AI Summary

This work addresses the lack of reliable and scalable evaluation benchmarks for large language models in reasoning over financial structured data. The authors propose an evaluation framework based on executable computation trees that automatically generates natural language questions from real-world financial tables and rigorously verifies answers through symbolic execution, enabling large-scale, high-quality dataset construction at zero annotation cost. The framework incorporates a four-dimensional controllable difficulty system—comprising computational depth, linguistic expressiveness, financial concept complexity, and context size—and integrates symbolic execution, natural language rendering, LoRA fine-tuning, and adversarial numerical perturbation testing. Experiments show model accuracy drops by up to 51% as reasoning depth increases and by 47% under adversarial perturbations; after LoRA fine-tuning, accuracy improves from 81.1% to 85.6% on the held-out test set and surpasses the FinQA baseline by 5 percentage points.

Abstract

While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.

Problem

• financial reasoning
• structured data
• benchmark
• LLM evaluation
• compositional reasoning

Innovation

• verified reasoning
• computation trees
• symbolic evaluation
• LoRA fine-tuning
• financial QA benchmark