From Lazy to Prolific: Tackling Missing Labels in Open Vocabulary Extreme Classification by Positive-Unlabeled Sequence Learning

2024-08-16

arXiv.org

AI Summary

To address model “lazy generation” and evaluation distortion caused by incomplete annotations in open-vocabulary extreme multi-label classification (OXMC), this paper proposes the Positive–Unlabeled Sequence Learning (PUSL) framework—the first to formulate OXMC as an infinite keyword generation task. Methodologically, PUSL integrates PU learning principles, dynamic decoding constraints, and multi-granularity evaluation techniques, specifically tailored for highly long-tailed texts in e-commerce and legal domains. A novel metric, B@k, is introduced and jointly deployed with F1@𝒪 to enable reliable evaluation under partial labeling. Experiments demonstrate that on severely skewed e-commerce data, PUSL discovers 30% more unique labels than baselines, with 72% of predictions aligning with real user queries; on EURLex-4.3K, it achieves significant F1 improvement, with performance gains becoming more pronounced as the label set expands to 30 classes.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Search and Optimization: Learning to Search
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Open-vocabulary Extreme Multi-label Classification (OXMC) extends traditional XMC by allowing prediction beyond an extremely large, predefined label set (typically $10^3$ to $10^{12}$ labels), addressing the dynamic nature of real-world labeling tasks. However, self-selection bias in data annotation leads to significant missing labels in both training and test data, particularly for less popular inputs. This creates two critical challenges: generation models learn to be"lazy'"by under-generating labels, and evaluation becomes unreliable due to insufficient annotation in the test set. In this work, we introduce Positive-Unlabeled Sequence Learning (PUSL), which reframes OXMC as an infinite keyphrase generation task, addressing the generation model's laziness. Additionally, we propose to adopt a suite of evaluation metrics, F1@$mathcal{O}$ and newly proposed B@$k$, to reliably assess OXMC models with incomplete ground truths. In a highly imbalanced e-commerce dataset with substantial missing labels, PUSL generates 30% more unique labels, and 72% of its predictions align with actual user queries. On the less skewed EURLex-4.3k dataset, PUSL demonstrates superior F1 scores, especially as label counts increase from 15 to 30. Our approach effectively tackles both the modeling and evaluation challenges in OXMC with missing labels.

Problem

• Open-Vocabulary Extreme Multi-Label Classification
• Lazy Model Issue
• Performance Evaluation Challenge

Innovation

• PUSL
• OXMC
• infinite keyword generation