From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating Financial Large Language Models

2025-08-18

AI Summary

Existing financial large language model (LLM) evaluations rely on single-scalar metrics, obscuring fine-grained capability disparities, and employ narrow, non-representative datasets that fail to reflect the true structure and limitations of financial knowledge. To address these issues, we propose FinCDM—the first cognitive diagnostic assessment framework for financial LLMs—introducing cognitive diagnosis theory to this domain for the first time. FinCDM establishes a fine-grained, expert-annotated skill taxonomy (the CPA-QKA dataset), enabling interpretable inference at both knowledge and skill levels. Through multidimensional response pattern analysis and modeling, FinCDM identifies critical yet previously underestimated deficiencies—such as tax reasoning and regulatory compliance inference—in 30 mainstream LLMs. It further uncovers systematic behavioral clustering patterns across models. By enhancing assessment transparency and diagnostic utility, FinCDM supports trustworthy, targeted development of financial LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large Language Models (LLMs) have shown promise for financial applications, yet their suitability for this high-stakes domain remains largely unproven due to inadequacies in existing benchmarks. Existing benchmarks solely rely on score-level evaluation, summarizing performance with a single score that obscures the nuanced understanding of what models truly know and their precise limitations. They also rely on datasets that cover only a narrow subset of financial concepts, while overlooking other essentials for real-world applications. To address these gaps, we introduce FinCDM, the first cognitive diagnosis evaluation framework tailored for financial LLMs, enabling the evaluation of LLMs at the knowledge-skill level, identifying what financial skills and knowledge they have or lack based on their response patterns across skill-tagged tasks, rather than a single aggregated number. We construct CPA-QKA, the first cognitively informed financial evaluation dataset derived from the Certified Public Accountant (CPA) examination, with comprehensive coverage of real-world accounting and financial skills. It is rigorously annotated by domain experts, who author, validate, and annotate questions with high inter-annotator agreement and fine-grained knowledge labels. Our extensive experiments on 30 proprietary, open-source, and domain-specific LLMs show that FinCDM reveals hidden knowledge gaps, identifies under-tested areas such as tax and regulatory reasoning overlooked by traditional benchmarks, and uncovers behavioral clusters among models. FinCDM introduces a new paradigm for financial LLM evaluation by enabling interpretable, skill-aware diagnosis that supports more trustworthy and targeted model development, and all datasets and evaluation scripts will be publicly released to support further research.

Problem

• Evaluating financial LLMs beyond single score metrics
• Identifying specific knowledge gaps in financial skills
• Addressing narrow coverage of existing financial benchmarks

Innovation

• Cognitive diagnosis framework for financial LLMs
• Skill-tagged evaluation instead of aggregated scores
• CPA-derived dataset with expert annotations