Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

2026-07-11

AI Summary

This study addresses the pronounced vulnerability of large language models in non-English and low-resource languages, where harmful prompts rejected in English are frequently executed incorrectly in other languages. The authors construct a multilingual jailbreaking benchmark spanning 18 languages across four resource tiers and incorporating four perturbation types—including script variation, code-switching, and translationese—and combine geometric mechanism analysis with subspace projection to systematically investigate how such perturbations affect safety alignment. Their findings reveal a sharp performance cliff between resource tiers 2 and 3 across all models and identify geometrically misaligned subspaces in low-resource languages that evade safety rejection mechanisms. These results demonstrate that reliance on English-only safety evaluations is insufficient and underscore the necessity of language-specific alignment strategies tailored to script families and perturbation categories.

Abstract

Safety alignment in large language models remains brittle across languages: prompts reliably refused in English can elicit harmful compliance in non-English and low-resource settings. We introduce \textsc{Minionese}, a multilingual jailbreak benchmark spanning 18 languages, 4 resource tiers, and 4 perturbation types (standard translation, code-switching, transliteration, and translationese), paired with a geometric mechanistic analysis of refusal failure across language tiers. We show that each attack type produces a distinct vulnerability profile: transliteration vulnerability is mediated by script identity, code-switching maintains effectiveness through the lowest-resource tier, and a sharp safety regime transition between Tiers 2 and 3 is consistent across all models. Mechanistically, low-resource jailbreaks succeed by routing harmful content through a geometrically misaligned subspace that projects insufficiently onto the refusal directions, leaving the refusal mechanism intact but untriggered. These findings show that English-only safety evaluations are insufficient; they require accounting for script family, perturbation type, and per-language alignment coverage. The benchmark and analysis code is at https://github.com/Brentkong/Minionese-Comprehensive-Benchmark-and-Mechanistic-Study-of-Multilingual-LLM-Safety.git.

Problem

• multilingual LLM safety
• jailbreak
• low-resource languages
• safety alignment
• refusal failure

Innovation

• multilingual LLM safety
• jailbreak benchmark
• geometric mechanistic analysis
• refusal subspace
• low-resource languages