An Efficient and Privacy-Preserving Architecture for Cross-Institutional Collaborative RAG

2026-05-25

AI Summary

This work addresses the challenge of deploying retrieval-augmented generation (RAG) systems across institutions constrained by data privacy regulations, which create data silos and hinder cross-node access to key-value caches in conventional Transformer self-attention mechanisms. To overcome these limitations, the authors propose FedRAG, a federated RAG framework featuring a novel Scrambled Distributed Attention protocol. This protocol integrates numerically stable feature obfuscation with dynamic token permutation, enabling high-throughput, privacy-preserving cross-institutional knowledge collaboration without requiring specialized hardware or model retraining. By decoupling attention computation from data locality, FedRAG effectively mitigates intermediate-state inversion attacks while incurring negligible utility loss (<0.1%). Empirical results demonstrate up to a 62× reduction in latency compared to existing secure baselines, achieving throughput sufficient for real-time human-readable inference.

Abstract

Retrieval-Augmented Generation (RAG) empowers LLMs with external knowledge, making cross-institutional domain-specific knowledge base integration a highly promising deployment paradigm. Despite this potential, strict privacy regulations create severe "data silos" that obstruct such collaboration. Building federated RAG systems requires distributed inference, but the Transformer's self-attention mechanism fundamentally conflicts with this by mandating cross-node access to distributed Key-Value caches. To address this challenge, we present FedRAG, a high-throughput, privacy-preserving federated RAG framework. At its core is a novel Scrambled Distributed Attention protocol that utilizes numerically stable feature scrambling and token permutation. By dynamically delegating scrambled computations to collaborating nodes, our system successfully decouples attention execution from data localization without exposing plaintext. Crucially, our approach requires no specialized hardware or model retraining, circumventing the prohibitive latency and communication overheads of cryptographic solutions while robustly defending against intermediate state inversion attacks. Extensive evaluations demonstrate our framework preserves negligible (<0.1\%) model utility degradation and achieves up to a 62$\times$ latency reduction over existing secure baselines, sustaining practical, human-reading throughput for cross-institutional knowledge synergy.

Problem

• Cross-Institutional Collaboration
• Privacy Preservation
• Retrieval-Augmented Generation
• Data Silos
• Distributed Inference

Innovation

• Federated RAG
• Scrambled Distributed Attention
• Privacy-Preserving Inference
• Cross-Institutional Collaboration
• Transformer Decoupling