JFinTEB: Japanese Financial Text Embedding Benchmark

2026-04-17

AI Summary

This work addresses the lack of embedding evaluation benchmarks tailored to Japanese financial texts, which hinders accurate assessment of model performance in this domain. To bridge this gap, we introduce JFinTEB, the first comprehensive benchmark specifically designed for Japanese financial text embeddings. JFinTEB encompasses retrieval and classification tasks, integrating representative scenarios such as instruction following, sentiment analysis, and document classification, along with a domain-adapted evaluation protocol. We systematically evaluate a range of embedding models—including Japanese-specific, multilingual, and commercial offerings—and publicly release the dataset and evaluation toolkit. This resource establishes a standardized evaluation framework for Japanese financial NLP research, effectively filling a critical void in available benchmarks.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We introduce JFinTEB, the first comprehensive benchmark specifically designed for evaluating Japanese financial text embeddings. Existing embedding benchmarks provide limited coverage of language-specific and domain-specific aspects found in Japanese financial texts. Our benchmark encompasses diverse task categories including retrieval and classification tasks that reflect realistic and well-defined financial text processing scenarios. The retrieval tasks leverage instruction-following datasets and financial text generation queries, while classification tasks cover sentiment analysis, document categorization, and domain-specific classification challenges derived from economic survey data. We conduct extensive evaluations across a wide range of embedding models, including Japanese-specific models of various sizes, multilingual models, and commercial embedding services. We publicly release JFinTEB datasets and evaluation framework at https://github.com/retarfi/JFinTEB to facilitate future research and provide a standardized evaluation protocol for the Japanese financial text mining community. This work addresses a critical gap in Japanese financial text processing resources and establishes a foundation for advancing domain-specific embedding research.

Problem

• Japanese financial text
• text embedding
• evaluation benchmark
• domain-specific
• language-specific

Innovation

• Japanese financial text embedding
• benchmark
• retrieval
• classification
• domain-specific evaluation