FinReflectKG: Agentic Construction and Evaluation of Financial Knowledge Graphs

2025-08-25

AI Summary

The financial domain lacks large-scale, open-source, semantically rich structured knowledge graphs (KGs), primarily due to the structural complexity of regulatory documents (e.g., annual reports) and stringent compliance requirements. Method: We introduce the first open-source financial KG derived from the latest SEC 10-K filings of S&P 100 companies. Our end-to-end pipeline features a table-aware chunking strategy and a schema-guided iterative extraction framework, augmented by a novel reflection-driven feedback mechanism and an LLM-as-a-Judge multi-dimensional evaluation system integrating intelligent parsing, rule-based validation, statistical verification, and large language model assessment. Contribution/Results: Experimental results demonstrate that our reflection-enabled agent achieves a compliance score of 64.8%, substantially outperforming single- and multi-step extraction baselines. It also attains state-of-the-art performance in precision, comprehensiveness, and relevance—marking significant advances in both accuracy and regulatory alignment for financial KG construction.

Technology Category

• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Natural Language Processing: Information Extraction
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The financial domain poses unique challenges for knowledge graph (KG) construction at scale due to the complexity and regulatory nature of financial documents. Despite the critical importance of structured financial knowledge, the field lacks large-scale, open-source datasets capturing rich semantic relationships from corporate disclosures. We introduce an open-source, large-scale financial knowledge graph dataset built from the latest annual SEC 10-K filings of all S and P 100 companies - a comprehensive resource designed to catalyze research in financial AI. We propose a robust and generalizable knowledge graph (KG) construction framework that integrates intelligent document parsing, table-aware chunking, and schema-guided iterative extraction with a reflection-driven feedback loop. Our system incorporates a comprehensive evaluation pipeline, combining rule-based checks, statistical validation, and LLM-as-a-Judge assessments to holistically measure extraction quality. We support three extraction modes - single-pass, multi-pass, and reflection-agent-based - allowing flexible trade-offs between efficiency, accuracy, and reliability based on user requirements. Empirical evaluations demonstrate that the reflection-agent-based mode consistently achieves the best balance, attaining a 64.8 percent compliance score against all rule-based policies (CheckRules) and outperforming baseline methods (single-pass and multi-pass) across key metrics such as precision, comprehensiveness, and relevance in LLM-guided evaluations.

Problem

• Constructing scalable financial knowledge graphs from complex documents
• Addressing lack of open-source datasets for financial semantic relationships
• Developing evaluation framework for extraction quality and compliance

Innovation

• Intelligent document parsing with table-aware chunking
• Schema-guided iterative extraction with reflection feedback
• Three extraction modes balancing efficiency and accuracy