Mitigating Language Bias in Cross-Lingual Job Retrieval: A Recruitment Platform Perspective

2025-02-05

AI Summary

In online recruitment platforms, multi-component text understanding—spanning job titles, responsibilities, and skill requirements—faces challenges including severe language bias, poor cross-lingual generalization, and fragmented modeling. To address these, we propose a multitask dual-encoder framework that jointly models heterogeneous textual components at multiple granularities. We further introduce the Language Bias KL Divergence (LBKL) metric to quantitatively assess language bias across domains. By integrating contrastive learning with multitask optimization, our model achieves superior cross-lingual job retrieval performance while maintaining lightweight architecture (reduced parameter count). Extensive experiments on multiple cross-lingual benchmarks demonstrate state-of-the-art results: LBKL decreases by 32.7%, and average retrieval accuracy improves by 5.8% over prior methods.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Understanding the textual components of resumes and job postings is critical for improving job-matching accuracy and optimizing job search systems in online recruitment platforms. However, existing works primarily focus on analyzing individual components within this information, requiring multiple specialized tools to analyze each aspect. Such disjointed methods could potentially hinder overall generalizability in recruitment-related text processing. Therefore, we propose a unified sentence encoder that utilized multi-task dual-encoder framework for jointly learning multiple component into the unified sentence encoder. The results show that our method outperforms other state-of-the-art models, despite its smaller model size. Moreover, we propose a novel metric, Language Bias Kullback-Leibler Divergence (LBKL), to evaluate language bias in the encoder, demonstrating significant bias reduction and superior cross-lingual performance.

Problem

• Reduces language bias in job retrieval
• Unifies resume and job posting analysis
• Improves cross-lingual recruitment accuracy

Innovation

• Unified sentence encoder application
• Multi-task dual-encoder framework usage
• LBKL metric for bias evaluation