Can Open-Weight Models Compete on Financial Text Comprehension?
This study addresses the lack of systematic evaluation of open-source large language models on real-world financial text understanding tasks by extending the Financial Touchstone benchmark, which now comprises 2,967 question-answer triplets derived from 495 international annual reports. The authors conduct a comprehensive assessment of 20 prominent models, including GLM-4.7, GLM-5, Kimi K2.6, DeepSeek V3.2, and leading closed-source counterparts. Notably, they find that certain non-reasoning open-source models—such as Kimi K2.6 and GLM-5—can match or even surpass some closed-source models in financial comprehension. The study also reveals that Chinese-language models often unjustifiably refuse legitimate queries due to content filtering mechanisms, with refusal behavior influenced by access pathways. Among evaluated models, Claude Opus 4.6 achieves the highest accuracy (88.4%), while Gemini 2.5 Pro exhibits the lowest hallucination rate (0.08%). Information retrieval errors account for 48.9% of failures. The full dataset and evaluation framework are publicly released.