🤖 AI Summary
This work addresses the challenge of evidence provenance errors in financial question answering, which frequently arise due to similar disclosures across sections, reporting periods, and companies in SEC filings. The authors introduce the first evidence-oriented benchmark for 10-K/10-Q–based financial QA and retrieval, requiring systems to precisely align entities, reporting periods, and contextual cues to locate supporting evidence. They contribute a novel set of human-annotated hard negatives—including confounding passages—and define three distinct evaluation tasks: retrieval, re-ranking, and hard negative discrimination. Evaluated on 1,185 annotated instances, even the strongest 7B-scale model achieves only 44.8% Recall@10 using BM25, instruction-tuned embeddings, and pairwise re-ranking. Performance drops by 13.0–20.5 percentage points when hard negatives are introduced, underscoring the task’s difficulty.
📝 Abstract
Financial question answering is typically evaluated by answer correctness, yet in SEC filings a plausible and even numerically correct answer can be grounded in the wrong evidence. Similar facts and disclosures recur across sections of a filing, across reporting periods of the same firm, and across comparable firms. FinRank targets this provenance-sensitive retrieval problem by requiring systems to identify evidence for the intended entity, reporting period, and disclosure context. The benchmark contains 1185 manually authored question-answer records over the 10-K and 10-Q filings of 22 companies. Each record includes a reference answer, gold supporting passages, and hand-curated hard negatives drawn from confusable passages within filings, across reporting periods, and across comparable firms. FinRank evaluates passage retrieval, reranking, and hard-negative discrimination as separately measured tasks. Baseline results demonstrate the difficulty of this setting: among the evaluated systems, even a 7B instruction-tuned embedder reaches only 44.8% Recall@10 on the pooled evidence corpus; sub-billion-parameter encoders gain at most 3.5 points over BM25, a finance-adapted embedder trails BM25 by 9.7 points, and pairwise accuracy falls by 13.0-20.5 percentage points when random negatives are replaced with the curated hard negatives. FinRank provides an evidence-first benchmark for developing financial question answering systems that are not only accurate but also grounded in the correct disclosure.