VaultGemma: A Differentially Private Gemma Model

2025-10-15

AI Summary

This work addresses privacy leakage risks in large language model (LLM) training by presenting the first end-to-end differentially private (DP) pretraining of a Gemma-series model. We train VaultGemma-1B—a 1-billion-parameter variant—using DP-SGD with gradient clipping and calibrated Gaussian noise injection, under strict ε-differential privacy guarantees (ε ≤ 8), on the same data mixture as Gemma 2. Unlike prior efforts focusing only on fine-tuning, our approach ensures DP compliance across the entire pretraining pipeline. Evaluation across multiple standard benchmarks shows that VaultGemma-1B retains performance close to its non-private counterpart, with an average degradation of less than 3.5%. To our knowledge, it is the first open-source Gemma variant satisfying end-to-end ε-DP. This work advances the practical deployment of privacy-preserving LLMs and releases all code, model weights, and training configurations to establish a reproducible benchmark for trustworthy LLM research.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We introduce VaultGemma 1B, a 1 billion parameter model within the Gemma family, fully trained with differential privacy. Pretrained on the identical data mixture used for the Gemma 2 series, VaultGemma 1B represents a significant step forward in privacy-preserving large language models. We openly release this model to the community

Problem

• Developing differentially private large language models
• Training billion-parameter models with privacy guarantees
• Advancing privacy-preserving capabilities in Gemma model family

Innovation

• Fully trained with differential privacy
• Based on 1 billion parameter Gemma model
• Pretrained on Gemma 2 series data