The Geopolitics of AI Safety: A Causal Analysis of Regional LLM Bias

2026-05-06

AI Summary

This study addresses the challenge of accurately assessing demographic bias in large language models (LLMs), noting that conventional fairness evaluations are often confounded by inherent toxicity in test data, leading to inflated bias estimates. To overcome this limitation, the work introduces causal inference into LLM safety auditing by proposing a probabilistic graphical model framework that leverages Pearl’s do-operator to intervene on cultural and demographic variables, thereby isolating their causal effects on model outputs. Empirical analyses across diverse models—including Llama, Qwen, and Mistral—and datasets such as ToxiGen and BOLD demonstrate that observational metrics substantially overstate bias, whereas causal interventions reveal that Western models exhibit higher refusal rates toward specific demographic groups, while Eastern models show lower overall intervention rates but heightened sensitivity to local populations. These findings underscore the inadequacy of traditional fairness metrics in capturing true causal disparities.

Abstract

As Large Language Models (LLMs) are integrated into global software systems, ensuring equitable safety guardrails is a critical requirement. Current fairness evaluations predominantly measure bias observationally, a methodology confounded by the inherent toxicity of topics naturally paired with specific demographics in testing datasets. This study introduces a Probabilistic Graphical Model (PGM) framework to audit LLM safety mechanisms causally. By applying Pearl's do-operator, we mathematically isolate the causal effect of injecting a cultural demographic into a prompt. We conduct a large-scale empirical analysis across seven instruction-tuned models spanning diverse origins: the United States (Llama-3.1-8B, Gemma-2-9B), Europe (Mistral-7B-v0.3), the UAE (Falcon3-7B), China (Qwen2.5-7B, DeepSeek-7B), and India (Airavata-7B). Utilizing two distinct datasets (ToxiGen and BOLD), the findings reveal a disparity between observational and interventional bias, demonstrating that standard fairness metrics can overestimate demographic bias by failing to account for context toxicity. Furthermore, the causal probabilities indicate distinct alignment trends: Western models exhibit higher causal refusal rates for specific demographic groups, whereas Eastern models demonstrate low overall intervention rates with targeted sensitivities toward regional demographics. We discuss the implications of these biases, highlighting how demographic-sensitive over-triggering restricts benign discourse in downstream applications.

Problem

• AI safety
• LLM bias
• fairness evaluation
• demographic bias
• geopolitics

Innovation

• causal inference
• probabilistic graphical model
• LLM safety
• demographic bias
• do-operator