Overview of the TalentCLEF 2025: Skill and Job Title Intelligence for Human Capital Management

2025-07-17

AI Summary

The human resource management domain has long lacked reliable, fair, and evaluable language intelligence models and open benchmarks. Method: This paper introduces the first multilingual public evaluation framework for skill–job intelligence, releasing a real-world, anonymized, human-annotated multilingual dataset with explicit gender bias analysis. Our approach integrates contrastive learning fine-tuning of multilingual encoders, information retrieval, large language model–based data augmentation, and re-ranking—supporting both monolingual and cross-lingual job matching and skill prediction. Contribution/Results: The benchmark attracted 76 teams submitting over 280 solutions; empirical analysis reveals that training strategies exert a significantly greater impact on performance than model scale. This work fills critical gaps in fairness, cross-lingual transferability, and reproducibility for labor-market language technologies and establishes the field’s first authoritative, open benchmark.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Advances in natural language processing and large language models are driving a major transformation in Human Capital Management, with a growing interest in building smart systems based on language technologies for talent acquisition, upskilling strategies, and workforce planning. However, the adoption and progress of these technologies critically depend on the development of reliable and fair models, properly evaluated on public data and open benchmarks, which have so far been unavailable in this domain. To address this gap, we present TalentCLEF 2025, the first evaluation campaign focused on skill and job title intelligence. The lab consists of two tasks: Task A - Multilingual Job Title Matching, covering English, Spanish, German, and Chinese; and Task B - Job Title-Based Skill Prediction, in English. Both corpora were built from real job applications, carefully anonymized, and manually annotated to reflect the complexity and diversity of real-world labor market data, including linguistic variability and gender-marked expressions. The evaluations included monolingual and cross-lingual scenarios and covered the evaluation of gender bias. TalentCLEF attracted 76 registered teams with more than 280 submissions. Most systems relied on information retrieval techniques built with multilingual encoder-based models fine-tuned with contrastive learning, and several of them incorporated large language models for data augmentation or re-ranking. The results show that the training strategies have a larger effect than the size of the model alone. TalentCLEF provides the first public benchmark in this field and encourages the development of robust, fair, and transferable language technologies for the labor market.

Problem

• Develop reliable models for skill and job title intelligence
• Address lack of public benchmarks in human capital management
• Evaluate multilingual job matching and skill prediction fairness

Innovation

• Multilingual encoder-based models fine-tuned
• Contrastive learning for model optimization
• Large language models for data augmentation