Uncovering Factor Level Preferences to Improve Human-Model Alignment

2024-10-09

arXiv.org

AI Summary

Large language models (LLMs) exhibit systematic misalignments with human preferences—e.g., verbosity and stylistic biases—yet existing evaluation methods remain coarse-grained and lack interpretability. Method: We propose PROFILE, the first framework for factor-level preference interpretability, integrating controllable prompt engineering, multidimensional preference disentanglement, factor attribution, and controlled comparative experiments across summarization, helpful response generation, and document question answering. Contribution/Results: PROFILE uncovers, for the first time, a “generation–evaluation gap”: human and LLM preferences diverge significantly during generation but align closely during evaluation. Leveraging this insight, we introduce a novel alignment strategy—either correcting misaligned preference factors or transferring evaluation-aligned strengths—which empirically improves alignment: +12.3% factual consistency in summarization and +9.7% helpfulness in response generation.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Despite advancements in Large Language Model (LLM) alignment, understanding the reasons behind LLM preferences remains crucial for bridging the gap between desired and actual behavior. LLMs often exhibit biases or tendencies that diverge from human preferences, such as favoring certain writing styles or producing overly verbose outputs. However, current methods for evaluating preference alignment often lack explainability, relying on coarse-grained comparisons. To address this, we introduce PROFILE (PRObing Factors of InfLuence for Explainability), a novel framework that uncovers and quantifies the influence of specific factors driving preferences. PROFILE's factor level analysis explains the 'why' behind human-model alignment and misalignment, offering insights into the direction of model improvement. We apply PROFILE to analyze human and LLM preferences across three tasks: summarization, helpful response generation, and document-based question-answering. Our factor level analysis reveals a substantial discrepancy between human and LLM preferences in generation tasks, whereas LLMs show strong alignment with human preferences in evaluation tasks. We demonstrate how leveraging factor level insights, including addressing misaligned factors or exploiting the generation-evaluation gap, can improve alignment with human preferences. This work underscores the importance of explainable preference analysis and highlights PROFILE's potential to provide valuable training signals, driving further improvements in human-model alignment.

Problem

• Identifying factor-level preference misalignments between humans and language models
• Developing automated framework to measure human-model preference alignment
• Addressing generation-discrimination gap to improve LLM-human alignment

Innovation

• Automated framework uncovers factor-level preference alignment
• Analyzes generation-discrimination gap to identify misalignments
• Improves alignment through fine-tuning with self-guidance