Towards Aligning Language Models with Textual Feedback

2024-07-24

Conference on Empirical Methods in Natural Language Processing

AI Summary

This work addresses the inefficiency, reliance on reinforcement learning (RL), and dependence on human annotations in aligning language models with user textual preferences. We propose ALT—a lightweight, natural-language-feedback-driven alignment paradigm that operates entirely without RL or explicit preference labels. ALT employs parameter-efficient fine-tuning based on conditional language modeling, eliminating reward modeling, policy gradients, and auxiliary modules; alignment is achieved solely via free-form textual feedback from users. In toxicity mitigation, ALT significantly outperforms PPO; in summarization and dialogue response generation, it matches PPO’s performance using only 20% of the training samples. Moreover, ALT enables closed-loop alignment via self-generated feedback from large language models. To our knowledge, this is the first end-to-end alignment method that fully leverages raw textual feedback—achieving both expressive flexibility and training simplicity without RL or explicit preference annotation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We present ALT (ALignment with Textual feedback), an approach that aligns language models with user preferences expressed in text. We argue that text offers greater expressiveness, enabling users to provide richer feedback than simple comparative preferences and this richer feedback can lead to more efficient and effective alignment. ALT aligns the model by conditioning its generation on the textual feedback. Our method relies solely on language modeling techniques and requires minimal hyper-parameter tuning, though it still presents the main benefit of RL-based algorithms and can effectively learn from textual feedback. We explore the efficacy and efficiency of textual feedback across different tasks such as toxicity reduction, summarization, and dialog response. We find that ALT outperforms PPO for the task of toxicity reduction while being able to match its performance on summarization with only 20% of the samples. We also explore how ALT can be used with feedback provided by an existing LLM.

Problem

• Aligns language models with user preferences using textual feedback.
• Improves model alignment efficiency and effectiveness through richer feedback.
• Explores efficacy of textual feedback in tasks like toxicity reduction and summarization.

Innovation

• Aligns models using textual feedback conditioning
• Utilizes language modeling without RL techniques
• Matches PPO performance with fewer samples