Differentially Private Retrieval-Augmented Generation

2026-02-16

AI Summary

This work addresses the privacy risks inherent in retrieval-augmented generation (RAG) systems when applied to sensitive domains such as healthcare and law, where database queries may inadvertently leak private information. To mitigate this, the authors propose DP-KSA, the first algorithm to integrate differential privacy into the RAG framework. Motivated by the observation that accurate answers often rely on only a few key terms, DP-KSA aggregates responses from multiple retrieved contexts and employs a differentially private mechanism to extract high-frequency keywords for constructing the final prompt. By combining keyword compression with the Propose-Test-Release paradigm, the method effectively curbs large language model hallucinations while providing rigorous formal privacy guarantees. Experimental results across two question-answering benchmarks and three instruction-tuned large language models demonstrate that DP-KSA achieves an excellent trade-off between privacy and utility, maintaining high answer accuracy and practicality under strong privacy protections.

Technology Category

• Machine Learning: Privacy
• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Retrieval-augmented generation (RAG) is a widely used framework for reducing hallucinations in large language models (LLMs) on domain-specific tasks by retrieving relevant documents from a database to support accurate responses. However, when the database contains sensitive corpora, such as medical records or legal documents, RAG poses serious privacy risks by potentially exposing private information through its outputs. Prior work has demonstrated that one can practically craft adversarial prompts that force an LLM to regurgitate the augmented contexts. A promising direction is to integrate differential privacy (DP), a privacy notion that offers strong formal guarantees, into RAG systems. However, naively applying DP mechanisms into existing systems often leads to significant utility degradation. Particularly for RAG systems, DP can reduce the usefulness of the augmented contexts leading to increase risk of hallucination from the LLMs. Motivated by these challenges, we present DP-KSA, a novel privacy-preserving RAG algorithm that integrates DP using the propose-test-release paradigm. DP-KSA follows from a key observation that most question-answering (QA) queries can be sufficiently answered with a few keywords. Hence, DP-KSA first obtains an ensemble of relevant contexts, each of which will be used to generate a response from an LLM. We utilize these responses to obtain the most frequent keywords in a differentially private manner. Lastly, the keywords are augmented into the prompt for the final output. This approach effectively compresses the semantic space while preserving both utility and privacy. We formally show that DP-KSA provides formal DP guarantees on the generated output with respect to the RAG database. We evaluate DP-KSA on two QA benchmarks using three instruction-tuned LLMs, and our empirical results demonstrate that DP-KSA achieves a strong privacy-utility tradeoff.

Problem

• Differential Privacy
• Retrieval-Augmented Generation
• Privacy Preservation
• Hallucination
• Sensitive Data

Innovation

• Differentially Private RAG
• Retrieval-Augmented Generation
• Privacy-Preserving LLM
• Propose-Test-Release
• Keyword-Based Augmentation