Improving Neutral Point of View Text Generation through Parameter-Efficient Reinforcement Learning and a Small-Scale High-Quality Dataset

2025-03-05

AI Summary

This study addresses the challenge of enhancing large language models’ (LLMs) ability to generate neutral-point-of-view (NPOV) text on sensitive topics. To this end, we introduce SHQ-NPOV—the first high-quality, human-annotated dataset comprising 300 sensitive-topic question-answer quadruples—and propose a parameter-efficient reinforcement learning (PE-RL) framework for NPOV optimization. Our methodology features a novel human-in-the-loop iterative peer-review and annotator-training paradigm, substantially improving annotation consistency and data quality. Experimental results demonstrate that PE-RL outperforms LoRA, supervised fine-tuning (SFT), and RLHF across all metrics under limited high-quality data: overall NPOV quality reaches 99.08% (+2.02 percentage points), supportive detail coverage increases by 24.96%, and oversimplification reduction improves by 22.69%. Moreover, the method exhibits strong cross-topic generalization, with no statistically significant performance degradation across domains.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper describes the construction of a dataset and the evaluation of training methods to improve generative large language models' (LLMs) ability to answer queries on sensitive topics with a Neutral Point of View (NPOV), i.e., to provide significantly more informative, diverse and impartial answers. The dataset, the SHQ-NPOV dataset, comprises 300 high-quality, human-written quadruplets: a query on a sensitive topic, an answer, an NPOV rating, and a set of links to source texts elaborating the various points of view. The first key contribution of this paper is a new methodology to create such datasets through iterative rounds of human peer-critique and annotator training, which we release alongside the dataset. The second key contribution is the identification of a highly effective training regime for parameter-efficient reinforcement learning (PE-RL) to improve NPOV generation. We compare and extensively evaluate PE-RL and multiple baselines-including LoRA finetuning (a strong baseline), SFT and RLHF. PE-RL not only improves on overall NPOV quality compared to the strongest baseline ($97.06% ightarrow 99.08%$), but also scores much higher on features linguists identify as key to separating good answers from the best answers ($60.25% ightarrow 85.21%$ for presence of supportive details, $68.74% ightarrow 91.43%$ for absence of oversimplification). A qualitative analysis corroborates this. Finally, our evaluation finds no statistical differences between results on topics that appear in the training dataset and those on separated evaluation topics, which provides strong evidence that our approach to training PE-RL exhibits very effective out of topic generalization.

Problem

• Improves LLMs' ability to generate neutral, informative, and impartial answers on sensitive topics.
• Introduces a high-quality dataset (SHQ-NPOV) for training and evaluating NPOV text generation.
• Proposes parameter-efficient reinforcement learning (PE-RL) to enhance NPOV generation quality.

Innovation

• Parameter-efficient reinforcement learning for NPOV generation
• SHQ-NPOV dataset with human-written quadruplets
• Iterative human peer-critique for dataset creation