P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs

2026-06-15

AI Summary

This study addresses the representational imbalance between European Portuguese (pt-PT) and Brazilian Portuguese (pt-BR) in current large language models, a gap compounded by the absence of systematic evaluation frameworks. To this end, the authors introduce P3B3—the first expert-curated, variant-neutral benchmark comprising multi-turn dialogue prompts alongside a dedicated evaluation framework—enabling fine-grained quantification of model preferences and controllability across Portuguese variants. Through human-annotated data and systematic experiments, the research reveals a pronounced and consistent bias toward pt-BR among mainstream large language models, while also uncovering substantial inter-model variation in variant controllability. These findings highlight a critical shortcoming in multilingual models’ capacity to equitably represent regional language varieties.

Abstract

As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored. To address this gap, we introduce P3B3, an expert-curated language variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability. Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.

Problem

• language variety bias
• Portuguese variants
• Large Language Models
• multilingual representation
• conversational benchmark

Innovation

• variety bias
• conversational benchmark
• Portuguese language varieties
• LLM controllability
• multilingual representation