Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

2026-07-07

AI Summary

Current AI safety evaluation frameworks are predominantly grounded in a Western-centric perspective, often overlooking regional legal, linguistic, and cultural differences, thereby introducing security vulnerabilities when deploying vision-language models globally. This work proposes the first multimodal, multilingual evaluation framework centered on cultural appropriateness, encompassing six Asia-Pacific countries and eight languages. By natively collecting localized image-text risk samples, the framework jointly surfaces both universally prohibited content and culturally sensitive issues. We introduce multimodal prompting strategies, construct an empirically driven cultural taxonomy, and integrate a Judge-Pluralis ensemble adjudication mechanism. Our findings reveal systemic model deficiencies in cross-cultural contexts—including image misinterpretation, lack of regional contextual awareness, and failure of refusal mechanisms—highlighting critical evaluation blind spots obscured by global aggregate metrics.

Abstract

Current AI safety evaluation and benchmarking frameworks predominantly rely on Western-centric culture-agnostic defaults that mask critical regional laws, socio-linguistic nuances, and cultural taboos, leaving Vision-Language Models (VLMs) vulnerable in global deployments. We introduce Pluralis v0.1: a novel multimodal, multi-regional, and multilingual dataset built from a culture-first perspective. Spanning 6,448 prompts across six Asia-Pacific countries (Bangladesh, India, Korea, Pakistan, Singapore, Taiwan) and eight languages, Pluralis diverges from prior work by natively sourcing localized safety hazards rather than adapting Western datasets. Crucially, it introduces a multimodal evaluation paradigm: user text (e.g., "Should I gift this?") and an image referring to "this" (e.g., a clock) - both innocuous in isolation, but synergistically triggering specific legal or cultural violations. Pluralis disentangles universal safety violations from localized cultural appropriateness, establishing the latter as a first-class evaluation axis. To operationalize this, we present Judge-Pluralis, an agreement-gated LLM-as-a-Judge ensemble trained on examples classified in an empirically derived cultural taxonomy. Observing VLM behavior on a subset of the Pluralis surfaces recurring, locale-specific failure modes such as image misidentifications with downstream harm, missed item-context-locale interactions, and inadequate refusals. These failure modes vary systematically across locales and languages, exposing blind spots that globally averaged metrics conceal. Ultimately, Pluralis is not presented as a solved evaluation framework for cultural alignment, but rather as a first step and catalyst for future innovation. We call upon the research community to utilize this foundation to advance the science of multilingual, multicultural evaluation to better support AI cultural alignment globally.

Problem

• AI safety
• cultural alignment
• multilingual evaluation
• multimodal benchmark
• Vision-Language Models

Innovation

• multicultural AI evaluation
• multimodal safety benchmark
• culture-first dataset
• localized AI alignment
• LLM-as-a-Judge ensemble