The Efficiency of Pre-training with Objective Masking in Pseudo Labeling for Semi-Supervised Text Classification

2025-05-10

AI Summary

To address low pseudo-label quality and poor generalization in few-shot semi-supervised text classification, this paper proposes a novel framework integrating target-masked unsupervised pretraining with teacher-student collaborative learning. Methodologically, it introduces target masking—previously unexplored in pseudo-labeling pretraining—to explicitly model class-distribution priors, thereby enhancing pseudo-label reliability and cross-lingual transferability. It further combines dynamic-threshold pseudo-label generation with bilingual consistency regularization over English and Swedish. Evaluated on three low-resource text classification benchmarks, the approach significantly outperforms baselines including Meta Pseudo Labels: under extreme few-shot settings (16–64 gold-labeled examples per class), it achieves up to a 4.2% absolute accuracy improvement. Results demonstrate superior effectiveness and robustness in ultra-low-resource scenarios, validating both the design rationale and practical utility of the proposed framework.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We extend and study a semi-supervised model for text classification proposed earlier by Hatefi et al. for classification tasks in which document classes are described by a small number of gold-labeled examples, while the majority of training examples is unlabeled. The model leverages the teacher-student architecture of Meta Pseudo Labels in which a ''teacher'' generates labels for originally unlabeled training data to train the ''student'' and updates its own model iteratively based on the performance of the student on the gold-labeled portion of the data. We extend the original model of Hatefi et al. by an unsupervised pre-training phase based on objective masking, and conduct in-depth performance evaluations of the original model, our extension, and various independent baselines. Experiments are performed using three different datasets in two different languages (English and Swedish).

Problem

• Improving semi-supervised text classification with few labeled examples
• Enhancing teacher-student model via objective masking pre-training
• Evaluating performance across multilingual datasets (English, Swedish)

Innovation

• Unsupervised pre-training with objective masking
• Teacher-student architecture for pseudo labeling
• Iterative model updates using gold-labeled data