Lessons from Defending Gemini Against Indirect Prompt Injections

2025-05-20

AI Summary

This work addresses the security risk of indirect prompt injection attacks against Gemini models during function calling and tool usage, stemming from integration with untrusted external data. We propose the first adaptive, continuous adversarial evaluation framework tailored for multi-generation Gemini models. Our method integrates dynamic sandbox monitoring, real-time tool permission enforcement, multi-round red-teaming exercises, and robustness quantification metrics—deeply embedding security testing into the model’s iterative development loop. Key innovations include: (1) a transferable, adaptive attack generation mechanism; and (2) fine-grained detection and blocking of malicious instructions within tool interaction chains. Experiments demonstrate that our framework significantly enhances Gemini’s resilience against indirect prompt injection, thereby advancing systematic security assurance for large language models in realistic tool-augmented deployment scenarios.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Gemini is increasingly used to perform tasks on behalf of users, where function-calling and tool-use capabilities enable the model to access user data. Some tools, however, require access to untrusted data introducing risk. Adversaries can embed malicious instructions in untrusted data which cause the model to deviate from the user's expectations and mishandle their data or permissions. In this report, we set out Google DeepMind's approach to evaluating the adversarial robustness of Gemini models and describe the main lessons learned from the process. We test how Gemini performs against a sophisticated adversary through an adversarial evaluation framework, which deploys a suite of adaptive attack techniques to run continuously against past, current, and future versions of Gemini. We describe how these ongoing evaluations directly help make Gemini more resilient against manipulation.

Problem

• Evaluating Gemini's robustness against indirect prompt injections
• Assessing risks from malicious instructions in untrusted data
• Improving model resilience through adversarial evaluation framework

Innovation

• Adversarial evaluation framework tests Gemini robustness
• Adaptive attack techniques simulate sophisticated adversaries
• Continuous evaluations enhance model resilience against manipulation