FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks

2026-04-10

AI Summary

Existing evaluations of financial tool usage by large language models are confined to single-step invocations, failing to capture trajectory-level reasoning capabilities over extended tasks. To address this gap, this work introduces FinTrace, a benchmark comprising 800 expert-annotated financial task trajectories spanning 34 real-world scenarios, along with FinTrace-Training—the first trajectory-level preference dataset for financial reasoning. A fine-grained evaluation protocol based on four dimensions and nine metrics is also proposed. Experiments reveal that while mainstream models exhibit competent tool selection, they suffer from notable deficiencies in reasoning coherence and final answer quality. Supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on Qwen-3.5-9B using FinTrace-Training significantly improve intermediate reasoning, with DPO more effectively suppressing failure modes; however, end-to-end answer accuracy remains a critical bottleneck.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Recent studies demonstrate that tool-calling capability enables large language models (LLMs) to interact with external environments for long-horizon financial tasks. While existing benchmarks have begun evaluating financial tool calling, they focus on limited scenarios and rely on call-level metrics that fail to capture trajectory-level reasoning quality. To address this gap, we introduce FinTrace, a benchmark comprising 800 expert-annotated trajectories spanning 34 real-world financial task categories across multiple difficulty levels. FinTrace employs a rubric-based evaluation protocol with nine metrics organized along four axes -- action correctness, execution efficiency, process quality, and output quality -- enabling fine-grained assessment of LLM tool-calling behavior. Our evaluation of 13 LLMs reveals that while frontier models achieve strong tool selection, all models struggle with information utilization and final answer quality, exposing a critical gap between invoking the right tools and reasoning effectively over their outputs. To move beyond diagnosis, we construct FinTrace-Training, the first trajectory-level preference dataset for financial tool-calling, containing 8,196 curated trajectories with tool-augmented contexts and preference pairs. We fine-tune Qwen-3.5-9B using supervised fine-tuning followed by direct preference optimization (DPO) and show that training on FinTrace-Training consistently improves intermediate reasoning metrics, with DPO more effectively suppressing failure modes. However, end-to-end answer quality remains a bottleneck, indicating that trajectory-level improvements do not yet fully propagate to final output quality.

Problem

• tool calling
• trajectory-level evaluation
• long-horizon financial tasks
• reasoning quality
• LLM evaluation

Innovation

• trajectory-level evaluation
• tool calling
• financial tasks
• preference dataset
• direct preference optimization