Multilingual hierarchical classification of job advertisements for job vacancy statistics

2024-11-06

AI Summary

This study addresses the challenge of accurately mapping multilingual job advertisements to both the International Standard Classification of Occupations (ISCO) and the Polish national classification (KZiS). We propose the first two-stage hierarchical Transformer model that explicitly incorporates occupational taxonomy structure. Methodologically, we construct a multilingual (24 languages) automated coding framework grounded in the six-digit ISCO-KZiS coding scheme, integrating a manually annotated central job repository, high-quality Polish–English machine translation, and conditional probability calibration. Our key contribution lies in embedding the tree-structured occupational hierarchy as architectural prior knowledge, substantially improving fine-grained coding consistency. Experiments demonstrate a 1–2 percentage point accuracy gain on human-coded job ads and strong cross-lingual generalization. We publicly release an open-source toolkit enabling reproducible, extensible occupational statistics and internationally comparable analysis.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The goal of this paper is to develop a multilingual classifier and conditional probability estimator of occupation codes for online job advertisements according in accordance with the International Standard Classification of Occupations (ISCO) extended with the Polish Classification of Occupations and Specializations (KZiS), which is analogous to the European Classification of Occupations. In this paper, we utilise a range of data sources, including a novel one, namely the Central Job Offers Database, which is a register of all vacancies submitted to Public Employment Offices. Their staff members code the vacancies according to the ISCO and KZiS. A hierarchical multi-class classifier has been developed based on the transformer architecture. The classifier begins by encoding the jobs found in advertisements to the widest 1-digit occupational group, and then narrows the assignment to a 6-digit occupation code. We show that incorporation of the hierarchical structure of occupations improves prediction accuracy by 1-2 percentage points, particularly for the hand-coded online job advertisements. Finally, a bilingual (Polish and English) and multilingual (24 languages) model is developed based on data translated using closed and open-source software. The open-source software is provided for the benefit of the official statistics community, with a particular focus on international comparability.

Problem

• Develop multilingual hierarchical classifier for job ad occupation codes
• Improve prediction accuracy using occupational hierarchy structure
• Create bilingual and multilingual models for international comparability

Innovation

• Transformer-based hierarchical classifier for job ads
• Multilingual model using translated data sources
• Open-source software for international comparability