Advanced Financial Reasoning at Scale: A Comprehensive Evaluation of Large Language Models on CFA Level III

2025-06-29

AI Summary

This study systematically evaluates the advanced financial reasoning capabilities of 23 large language models (LLMs) on the CFA Level III examination—a high-stakes, domain-specific benchmark requiring nuanced analytical and communicative proficiency. Method: We assess performance across both multiple-choice and constructed-response tasks, employing chain-of-thought and self-discovery prompting strategies; critically, we introduce the first rigorous, multi-dimensional scoring rubric specifically designed for professional finance-oriented essay questions. Contribution/Results: o4-mini and Gemini 2.5 Flash achieve top composite scores of 79.1% and 77.3%, respectively, demonstrating that contemporary LLMs possess practical utility for high-level financial reasoning. However, substantial performance gaps between constructed-response and multiple-choice tasks reveal persistent weaknesses in structured analysis, assumption identification, and domain-precise articulation. Our work establishes a reproducible, domain-grounded evaluation framework and provides empirical guidance for LLM selection in mission-critical financial applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

As financial institutions increasingly adopt Large Language Models (LLMs), rigorous domain-specific evaluation becomes critical for responsible deployment. This paper presents a comprehensive benchmark evaluating 23 state-of-the-art LLMs on the Chartered Financial Analyst (CFA) Level III exam - the gold standard for advanced financial reasoning. We assess both multiple-choice questions (MCQs) and essay-style responses using multiple prompting strategies including Chain-of-Thought and Self-Discover. Our evaluation reveals that leading models demonstrate strong capabilities, with composite scores such as 79.1% (o4-mini) and 77.3% (Gemini 2.5 Flash) on CFA Level III. These results, achieved under a revised, stricter essay grading methodology, indicate significant progress in LLM capabilities for high-stakes financial applications. Our findings provide crucial guidance for practitioners on model selection and highlight remaining challenges in cost-effective deployment and the need for nuanced interpretation of performance against professional benchmarks.

Problem

• Evaluating LLMs on advanced financial reasoning using CFA Level III exam.
• Assessing multiple-choice and essay responses with diverse prompting strategies.
• Identifying model capabilities and challenges for financial industry deployment.

Innovation

• Evaluates 23 LLMs on CFA Level III exam
• Uses Chain-of-Thought and Self-Discover prompting
• Revised strict essay grading methodology