The Effect of Multi-Lingual and Keyword Adversarial Injection on LLM Relevance Judgment

2026-07-10

AI Summary

This study addresses the vulnerability of large language models (LLMs) when employed as relevance assessors in multilingual information retrieval, revealing that their cross-lingual generalization capabilities can themselves serve as attack vectors. The authors systematically evaluate the impact of both instruction-based and content-based prompt injection attacks across eight languages with varying resource availability, using the TREC Deep Learning dataset, two open-source LLMs, and a standardized prompting framework. Their experiments demonstrate that multilingual query injections can significantly inflate relevance scores and effectively bypass existing defense mechanisms. Even adapted defenses remain susceptible to targeted adversarial variants, underscoring the urgent need for proactive and robust evaluation frameworks to safeguard LLM-based relevance judgments in multilingual settings.

Abstract

Large language models (LLMs) are increasingly being used as automated judges for relevance evaluation in information retrieval, yet their robustness to adversarial manipulation remains insufficiently understood, particularly in multilingual settings. In this work, we investigate the impact of cross-lingual prompt injection attacks on LLM-based relevance judgments using TREC Deep Learning collections and two open-weight models under established prompting frameworks. We examine both instruction-based and content-based injection strategies in 8 languages spanning different resource levels. Our results demonstrate that multilingual query-based injections are highly effective in inflating relevance scores while simultaneously evading existing prompt-injection defenses. We further found that, although existing defense mechanisms can be modified to mitigate such attacks, these injections can be easily adapted to bypass them. These findings highlight a critical gap in current defense approaches and demonstrate that language generalization can act as an attack vector, underscoring the need for more robust and proactive evaluation frameworks for LLM-as-a-judge systems.

Problem

• adversarial injection
• multilingual
• relevance judgment
• prompt injection
• LLM-as-a-judge

Innovation

• multilingual adversarial injection
• LLM-as-a-judge
• prompt injection attack
• cross-lingual robustness
• relevance judgment