Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

2026-02-16

AI Summary

This work addresses the gap in existing bias evaluation benchmarks, which lack coverage of gender and homophobic biases within the Philippine cultural context. The authors propose FilBBQ—the first bias benchmark tailored to Filipino sociocultural norms—constructed through a four-stage pipeline involving template categorization, culturally adapted translation, new template creation, and prompt generation, yielding over 10,000 evaluation samples. To enhance assessment robustness, they introduce a multi-random-seed response averaging mechanism. Experimental results reveal significant biases in mainstream Filipino-language question-answering models across dimensions such as emotional expression, domestic role allocation, stereotyping of queer interests, and polygamy. Notably, bias scores exhibit substantial variance across random seeds, underscoring FilBBQ’s novel contribution in advancing both cultural sensitivity and reliability in bias evaluation.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

With natural language generation becoming a popular use case for language models, the Bias Benchmark for Question-Answering (BBQ) has grown to be an important benchmark format for evaluating stereotypical associations exhibited by generative models. We expand the linguistic scope of BBQ and construct FilBBQ through a four-phase development process consisting of template categorization, culturally aware translation, new template construction, and prompt generation. These processes resulted in a bias test composed of more than 10,000 prompts which assess whether models demonstrate sexist and homophobic prejudices relevant to the Philippine context. We then apply FilBBQ on models trained in Filipino but do so with a robust evaluation protocol that improves upon the reliability and accuracy of previous BBQ implementations. Specifically, we account for models'response instability by obtaining prompt responses across multiple seeds and averaging the bias scores calculated from these distinctly seeded runs. Our results confirm both the variability of bias scores across different seeds and the presence of sexist and homophobic biases relating to emotion, domesticity, stereotyped queer interests, and polygamy. FilBBQ is available via GitHub.

Problem

• bias evaluation
• Filipino language models
• sexism
• homophobia
• question-answering

Innovation

• FilBBQ
• bias evaluation
• robust evaluation protocol
• culturally aware translation
• response instability