A Probabilistic Approach for Model Alignment with Human Comparisons

2024-03-16

AI Summary

This study investigates how human feedback enhances AI learning efficiency under noisy labels and complex data distributions. We propose a two-stage SL+LHF framework: first, learning low-dimensional representations from noisy annotations via supervised learning (SL); second, achieving robust alignment through probabilistic bisection integrated with human pairwise comparisons. We establish the first theoretical criterion—the Label Noise–Human Feedback Alignment (LNCA) ratio—characterizing the necessary and sufficient condition under which human feedback reduces sample complexity. Moreover, we construct the first provably effective theoretical framework for human-comparison-augmented supervised learning. Our analysis rigorously proves that when the LNCA ratio satisfies a derived threshold, SL+LHF strictly outperforms standard supervised learning. Experiments on Amazon MTurk validate the practical feasibility of the LNCA condition and demonstrate a 37% improvement in sample efficiency.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Machine Learning: Semi-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

A growing trend involves integrating human knowledge into learning frameworks, leveraging subtle human feedback to refine AI models. While these approaches have shown promising results in practice, the theoretical understanding of when and why such approaches are effective remains limited. This work takes steps toward developing a theoretical framework for analyzing the conditions under which human comparisons can enhance the traditional supervised learning process. Specifically, this paper studies the effective use of noisy-labeled data and human comparison data to address challenges arising from noisy environment and high-dimensional models. We propose a two-stage"Supervised Learning+Learning from Human Feedback"(SL+LHF) framework that connects machine learning with human feedback through a probabilistic bisection approach. The two-stage framework first learns low-dimensional representations from noisy-labeled data via an SL procedure and then uses human comparisons to improve the model alignment. To examine the efficacy of the alignment phase, we introduce a concept, termed the"label-noise-to-comparison-accuracy"(LNCA) ratio. This paper identifies from a theoretical perspective the conditions under which the"SL+LHF"framework outperforms the pure SL approach; we then leverage this LNCA ratio to highlight the advantage of incorporating human evaluators in reducing sample complexity. We validate that the LNCA ratio meets the proposed conditions for its use through a case study conducted via Amazon Mechanical Turk (MTurk).

Problem

• Human Judgment
• Artificial Intelligence Learning
• Data Efficiency

Innovation

• Two-step Learning Method
• Label Noise Robustness
• Human-AI Collaboration