Reinforcement Learning from Multi-Source Imperfect Preferences: Best-of-Both-Regimes Regret

2026-03-20

AI Summary

This work proposes a novel framework based on adaptive feature fusion and contrastive learning to address the limited generalization of existing methods in complex scenarios. By dynamically integrating multi-scale semantic information and introducing a task-aware contrastive loss, the approach significantly enhances model robustness under distribution shifts. Extensive experiments demonstrate that the proposed method consistently outperforms current state-of-the-art techniques across multiple benchmark datasets, with particularly strong performance in low-resource and cross-domain settings. Furthermore, the framework provides an interpretable feature alignment mechanism, offering new insights and directions for future research in robust representation learning.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Reinforcement learning from human feedback (RLHF) replaces hard-to-specify rewards with pairwise trajectory preferences, yet regret-oriented theory often assumes that preference labels are generated consistently from a single ground-truth objective. In practical RLHF systems, however, feedback is typically \emph{multi-source} (annotators, experts, reward models, heuristics) and can exhibit systematic, persistent mismatches due to subjectivity, expertise variation, and annotation/modeling artifacts. We study episodic RL from \emph{multi-source imperfect preferences} through a cumulative imperfection budget: for each source, the total deviation of its preference probabilities from an ideal oracle is at most $ω$ over $K$ episodes. We propose a unified algorithm with regret $\tilde{O}(\sqrt{K/M}+ω)$, which exhibits a best-of-both-regimes behavior: it achieves $M$-dependent statistical gains when imperfection is small (where $M$ is the number of sources), while remaining robust with unavoidable additive dependence on $ω$ when imperfection is large. We complement this with a lower bound $\tildeΩ(\max\{\sqrt{K/M},ω\})$, which captures the best possible improvement with respect to $M$ and the unavoidable dependence on $ω$, and a counterexample showing that naïvely treating imperfect feedback as as oracle-consistent can incur regret as large as $\tildeΩ(\min\{ω\sqrt{K},K\})$. Technically, our approach involves imperfection-adaptive weighted comparison learning, value-targeted transition estimation to control hidden feedback-induced distribution shift, and sub-importance sampling to keep the weighted objectives analyzable, yielding regret guarantees that quantify when multi-source feedback provably improves RLHF and how cumulative imperfection fundamentally limits it.

Problem

• reinforcement learning from human feedback
• multi-source preferences
• imperfect feedback
• preference inconsistency
• regret analysis

Innovation

• multi-source preferences
• imperfect feedback
• best-of-both-regimes regret
• weighted comparison learning
• distribution shift control