PolyLingua: Margin-based Inter-class Transformer for Robust Cross-domain Language Detection

2025-12-08

AI Summary

Low accuracy in cross-domain language identification (e.g., mismatch between song titles and user language in music requests), poor robustness of lightweight models (e.g., LangDetect, FastText), and high deployment costs of large language models (LLMs) hinder multilingual system performance. To address these challenges, this paper proposes PolyLingua—a lightweight Transformer-based language identification model. Its core innovation is a two-stage contrastive learning framework: instance-level separation enhances inter-class discriminability, while category-level alignment enforces semantic consistency across languages; an adaptive margin mechanism further improves fine-grained discrimination among typologically similar languages. Evaluated on the Amazon Massive and Song datasets, PolyLingua achieves 99.25% and 98.15% F1 scores, respectively—surpassing Sonnet 3.5 while reducing parameter count by 10×. The model delivers high accuracy, low inference latency, and strong adaptability to resource-constrained environments.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Language identification is a crucial first step in multilingual systems such as chatbots and virtual assistants, enabling linguistically and culturally accurate user experiences. Errors at this stage can cascade into downstream failures, setting a high bar for accuracy. Yet, existing language identification tools struggle with key cases--such as music requests where the song title and user language differ. Open-source tools like LangDetect, FastText are fast but less accurate, while large language models, though effective, are often too costly for low-latency or low-resource settings. We introduce PolyLingua, a lightweight Transformer-based model for in-domain language detection and fine-grained language classification. It employs a two-level contrastive learning framework combining instance-level separation and class-level alignment with adaptive margins, yielding compact and well-separated embeddings even for closely related languages. Evaluated on two challenging datasets--Amazon Massive (multilingual digital assistant utterances) and a Song dataset (music requests with frequent code-switching)--PolyLingua achieves 99.25% F1 and 98.15% F1, respectively, surpassing Sonnet 3.5 while using 10x fewer parameters, making it ideal for compute- and latency-constrained environments.

Problem

• Improves language detection accuracy in multilingual systems
• Addresses challenges with code-switching and ambiguous language cases
• Provides lightweight solution for low-resource and low-latency settings

Innovation

• Lightweight Transformer model for language detection
• Two-level contrastive learning with adaptive margins
• Compact embeddings for closely related languages