Align Generative Artificial Intelligence with Human Preferences: A Novel Large Language Model Fine-Tuning Method for Online Review Management

2026-04-22

AI Summary

This work addresses key challenges faced by general-purpose large language models in generating merchant responses to customer reviews, including hallucination, difficulty in modeling human preferences, and overly conservative response strategies. To tackle these issues, the authors propose a preference fine-tuning framework that integrates context augmentation, theory-driven construction of automatic preference pairs, curriculum learning, and support constraints derived from density estimation. This approach effectively mitigates hallucination and domain adaptation difficulties while preserving theoretical guarantees and enhancing both the quality and diversity of generated replies. Experimental results demonstrate that the proposed method significantly outperforms existing baselines, achieving notable improvements in alignment with human preferences, practical utility, and user satisfaction.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Online reviews have played a pivotal role in consumers' decision-making processes. Existing research has highlighted the significant impact of managerial review responses on customer relationship management and firm performance. However, a large portion of online reviews remains unaddressed due to the considerable human labor required to respond to the rapid growth of online reviews. While generative AI has achieved remarkable success in a range of tasks, they are general-purpose models and may not align well with domain-specific human preferences. To tailor these general generative AI models to domain-specific applications, finetuning is commonly employed. Nevertheless, several challenges persist in finetuning with domain-specific data, including hallucinations, difficulty in representing domain-specific human preferences, and over conservatism in offline policy optimization. To address these challenges, we propose a novel preference finetuning method to align an LLM with domain-specific human preferences for generating online review responses. Specifically, we first identify the source of hallucination and propose an effective context augmentation approach to mitigate the LLM hallucination. To represent human preferences, we propose a novel theory-driven preference finetuning approach that automatically constructs human preference pairs in the online review domain. Additionally, we propose a curriculum learning approach to further enhance preference finetuning. To overcome the challenge of over conservatism in existing offline preference finetuning method, we propose a novel density estimation-based support constraint method to relax the conservatism, and we mathematically prove its superior theoretical guarantees. Extensive evaluations substantiate the superiority of our proposed preference finetuning method.

Problem

• Generative Artificial Intelligence
• Human Preferences
• Large Language Model
• Online Review Management
• Fine-Tuning

Innovation

• preference fine-tuning
• hallucination mitigation
• context augmentation
• curriculum learning
• support constraint