IndoBERT-Relevancy: A Context-Conditioned Relevancy Classifier for Indonesian Text

2026-03-27

AI Summary

This work addresses the lack of effective models for Indonesian relevance classification by proposing a context-conditioned relevance classifier built upon IndoBERT Large (335 million parameters). Leveraging an iterative, failure-driven data construction methodology, the authors curate a novel dataset comprising 31,360 annotated sentence pairs spanning 188 topics, which integrates both authentic multi-source data and targeted synthetic examples to encompass both formal and informal Indonesian text. The resulting model achieves 96.5% accuracy and an F1 score of 0.948 on the test set, demonstrating substantially improved robustness. The model and dataset have been publicly released on the Hugging Face platform to support further research in Indonesian natural language processing.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Determining whether a piece of text is relevant to a given topic is a fundamental task in natural language processing, yet it remains largely unexplored for Bahasa Indonesia. Unlike sentiment analysis or named entity recognition, relevancy classification requires the model to reason about the relationship between two inputs simultaneously: a topical context and a candidate text. We introduce IndoBERT-Relevancy, a context-conditioned relevancy classifier built on IndoBERT Large (335M parameters) and trained on a novel dataset of 31,360 labeled pairs spanning 188 topics. Through an iterative, failure-driven data construction process, we demonstrate that no single data source is sufficient for robust relevancy classification, and that targeted synthetic data can effectively address specific model weaknesses. Our final model achieves an F1 score of 0.948 and an accuracy of 96.5%, handling both formal and informal Indonesian text. The model is publicly available at HuggingFace.

Problem

• relevancy classification
• Indonesian text
• topic relevance
• natural language processing

Innovation

• relevancy classification
• context-conditioned modeling
• failure-driven data construction
• synthetic data augmentation
• IndoBERT