Steerable Cultural Preference Optimization of Reward Models

2026-06-16

AI Summary

This work addresses the challenge that existing large language model alignment methods often rely on monolithic, region-specific preference data, failing to account for the diverse values of global cultural subgroups. To overcome this limitation, the authors propose Sensitivity-aware Cultural Preference Optimization (SCPO), an algorithm that dynamically balances multicultural preferences during reward model training through a tunable weighting mechanism. SCPO enables the first scalable and unbiased modeling of cross-national values, substantially reducing systemic bias against minority groups. Evaluated on multi-country annotated datasets—PRISM and GlobalOpinionQA—the method achieves up to a 7-percentage-point improvement in reward model performance for minority populations across seven countries, while offering a 280% gain in data efficiency compared to full fine-tuning.

Abstract

It is essential for large language model (LLM) technology to serve many different cultural sub-communities in a manner that is acceptable to each community. However, research on LLM alignment has so far predominantly focused on predicting a unified response preference of annotators from certain regions. This paper aims to advance the development of alignment models with a more global outlook, that are able to accurately represent the preferences of subcommunities and do not exhibit excessive bias towards any of them. We focus on the development of reward models for this purpose and present a novel reward model training algorithm (SCPO) that can incorporate diverse cultural preferences in a balanced manner. Our method results in performance increases of the minority reward model of up to 7 points over the baseline model across two datasets, PRISM and GlobalOpinionQA, and across 7 countries. SCPO is up to 280% more training data-efficient than full-data finetuning of reward models. In addition, we perform analysis of bias by separately evaluating on the preference of subcommunities and show that excessive bias is mitigated via our weighting method. Our code is available at https://github.com/minsik-ai/Steerable-Cultural-Preference

Problem

• cultural preference
• reward model
• LLM alignment
• bias mitigation
• subcommunity representation

Innovation

• Cultural Preference Optimization
• Reward Model
• LLM Alignment
• Bias Mitigation
• Data Efficiency