Naturally Occurring Feedback is Common, Extractable and Useful

2024-07-15

AI Summary

This study addresses the high cost and unsustainability of conventional human-annotated preference feedback for aligning language models with human preferences. To overcome this limitation, we propose a “natural feedback mining” paradigm: first, manual verification confirms that approximately 30% of real-world dialogues contain explicit user feedback; second, we develop a hybrid feedback identification model—combining rule-based heuristics and supervised fine-tuning—to automatically extract over 200,000 high-quality feedback instances from more than one million authentic human–model interactions. This work provides the first systematic empirical validation of both the prevalence and supervisory utility of naturally occurring feedback. Experiments demonstrate that preference models trained on this data significantly outperform strong baselines across multiple alignment benchmarks—including reward modeling, RLHF, and direct preference optimization—thereby establishing a low-resource, scalable, and sustainable pathway for language model alignment.

Technology Category

• Natural Language Processing: (Large) Language Models
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Human feedback data is a critical component in developing language models. However, collecting this feedback is costly and ultimately not scalable. Inspired by the way human interlocutors provide spontaneous unsolicited feedback to each other, we propose to extract feedback that users naturally include when interacting with chat models. We manually annotated conversations to confirm the presence of naturally occurring feedback in a standard corpus, finding that as much as 30% of the chats include explicit feedback. Comparing to older datasets, we find that naturally occurring feedback is more prevalent in recent conversation datasets, suggesting that more than ever, naturally occurring feedback can serve as a valuable resource for feedback data. We propose a method for automatically extracting this feedback, and apply it to over 1M conversations to obtain hundreds of thousands of feedback samples. The extracted feedback shows promise: training with it improves over baseline models and enhances model alignment to human preferences.

Problem

• Extracting naturally occurring feedback from user interactions.
• Reducing costs and improving scalability of feedback collection.
• Enhancing language models using extracted feedback data.

Innovation

• Extract naturally occurring feedback from chats
• Automate feedback extraction from 1M conversations
• Use extracted feedback to improve model alignment