Self Iterative Label Refinement via Robust Unlabeled Learning

2025-02-18

AI Summary

To address error accumulation in self-supervised classification with large language models (LLMs)—caused by high pseudo-label noise, inherent biases, overconfidence, and knowledge blind spots—this paper proposes the first iterative pseudo-label refinement framework based on unlabeled–unlabeled (UU) learning. Our method requires neither human annotations nor trusted validation sets; instead, it leverages two unlabeled datasets with differing positive-class proportions to enable continuous pseudo-label denoising and dual-distribution unsupervised calibration. By innovatively integrating UU learning into the LLM’s self-iteration pipeline, we effectively mitigate overconfidence and domain-knowledge deficiencies that exacerbate bias. Empirical evaluation across low-resource language classification, patent categorization, and protein structure identification demonstrates an average accuracy improvement of 5.2% over self-refinement results from GPT-4o and DeepSeek-R1.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advances in large language models (LLMs) have yielded impressive performance on various tasks, yet they often depend on high-quality feedback that can be costly. Self-refinement methods attempt to leverage LLMs' internal evaluation mechanisms with minimal human supervision; however, these approaches frequently suffer from inherent biases and overconfidence, especially in domains where the models lack sufficient internal knowledge, resulting in performance degradation. As an initial step toward enhancing self-refinement for broader applications, we introduce an iterative refinement pipeline that employs the Unlabeled-Unlabeled learning framework to improve LLM-generated pseudo-labels for classification tasks. By exploiting two unlabeled datasets with differing positive class ratios, our approach iteratively denoises and refines the initial pseudo-labels, thereby mitigating the adverse effects of internal biases with minimal human supervision. Evaluations on diverse datasets, including low-resource language corpora, patent classifications, and protein structure categorizations, demonstrate that our method consistently outperforms both initial LLM's classification performance and the self-refinement approaches by cutting-edge models (e.g., GPT-4o and DeepSeek-R1).

Problem

• Enhance self-refinement methods for LLMs
• Mitigate biases in pseudo-label generation
• Improve classification with unlabeled datasets

Innovation

• Iterative refinement pipeline
• Unlabeled-Unlabeled learning framework
• Mitigates internal biases minimally supervised