FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models

2025-05-31

AI Summary

To address three critical bottlenecks of large language models (LLMs) in finance—weak discriminative performance, inaccurate domain-specific retrieval, and inadequate topic modeling—this paper introduces FinBERT2, a finance-specialized bidirectional encoder. Methodologically, FinBERT2 features: (i) the first lightweight financial encoder architecture designed for the LLM era; (ii) the largest publicly available Chinese financial pretraining corpus to date (32B tokens); and (iii) unified support for discriminative classification, semantic retrieval, and topic modeling. It leverages financial self-supervised pretraining, task-adaptive fine-tuning (via classification and contrastive learning), domain-enhanced embedding learning, and headline-aware topic modeling. Empirically, FinBERT2 outperforms mainstream LLMs by 9.7–12.3% on five financial classification benchmarks, surpasses open-source and commercial embedding models by 4.2–6.8% in financial retrieval accuracy, and significantly improves clustering quality and topic representation fidelity for financial headlines.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In natural language processing (NLP), the focus has shifted from encoder-only tiny language models like BERT to decoder-only large language models(LLMs) such as GPT-3. However, LLMs' practical application in the financial sector has revealed three limitations: (1) LLMs often perform worse than fine-tuned BERT on discriminative tasks despite costing much higher computational resources, such as market sentiment analysis in financial reports; (2) Application on generative tasks heavily relies on retrieval augmented generation (RAG) methods to provide current and specialized information, with general retrievers showing suboptimal performance on domain-specific retrieval tasks; (3) There are additional inadequacies in other feature-based scenarios, such as topic modeling. We introduce FinBERT2, a specialized bidirectional encoder pretrained on a high-quality, financial-specific corpus of 32b tokens. This represents the largest known Chinese financial pretraining corpus for models of this parameter size. As a better backbone, FinBERT2 can bridge the gap in the financial-specific deployment of LLMs through the following achievements: (1) Discriminative fine-tuned models (Fin-Labelers) outperform other (Fin)BERT variants by 0.4%-3.3% and leading LLMs by 9.7%-12.3% on average across five financial classification tasks. (2) Contrastive fine-tuned models (Fin-Retrievers) outperform both open-source (e.g., +6.8% avg improvement over BGE-base-zh) and proprietary (e.g., +4.2% avg improvement over OpenAI's text-embedding-3-large) embedders across five financial retrieval tasks; (3) Building on FinBERT2 variants, we construct the Fin-TopicModel, which enables superior clustering and topic representation for financial titles. Our work revisits financial BERT models through comparative analysis with contemporary LLMs and offers practical insights for effectively utilizing FinBERT in the LLMs era.

Problem

• LLMs underperform BERT in financial discriminative tasks despite higher costs
• Domain-specific retrieval tasks show suboptimal performance with general retrievers
• Existing models lack adequacy in financial feature-based scenarios like topic modeling

Innovation

• Specialized bidirectional encoder for finance
• Pretrained on 32b financial tokens
• Outperforms BERT and LLMs in finance