Safeguarding LLM Embeddings in End-Cloud Collaboration via Entropy-Driven Perturbation

2025-03-17

AI Summary

In edge-cloud collaborative systems, embedding inversion attacks (EIA) can reconstruct original text from embeddings, posing severe privacy risks. This paper proposes EntroGuard, an entropy-driven perturbation method that defends against EIA by applying controllable, model-agnostic perturbations to edge-side text embeddings—without modifying the local LLM’s embedding model. Its core contribution is the first formulation of **Shannon entropy of reconstructed text** as the optimization objective, jointly maximizing reconstruction entropy under bounded ℓ₂ perturbation constraints and adaptive allocation of perturbations across sparse/redundant regions of the embedding space—enabling fine-grained privacy-utility trade-offs. EntroGuard is plug-and-play and lightweight for edge deployment. Experiments show it reduces EIA-induced privacy leakage risk by up to 8×, with negligible impact on retrieval accuracy, significantly outperforming existing embedding privacy protection methods.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent studies improve on-device language model (LM) inference through end-cloud collaboration, where the end device retrieves useful information from cloud databases to enhance local processing, known as Retrieval-Augmented Generation (RAG). Typically, to retrieve information from the cloud while safeguarding privacy, the end device transforms original data into embeddings with a local embedding model. However, the recently emerging Embedding Inversion Attacks (EIAs) can still recover the original data from text embeddings (e.g., training a recovery model to map embeddings back to original texts), posing a significant threat to user privacy. To address this risk, we propose EntroGuard, an entropy-driven perturbation-based embedding privacy protection method, which can protect the privacy of text embeddings while maintaining retrieval accuracy during the end-cloud collaboration. Specifically, to defeat various EIAs, we perturb the embeddings to increase the entropy of the recovered text in the common structure of recovery models, thus steering the embeddings toward meaningless texts rather than original sensitive texts during the recovery process. To maintain retrieval performance in the cloud, we constrain the perturbations within a bound, applying the strategy of reducing them where redundant and increasing them where sparse. Moreover, EntroGuard can be directly integrated into end devices without requiring any modifications to the embedding model. Extensive experimental results demonstrate that EntroGuard can reduce the risk of privacy leakage by up to 8 times at most with negligible loss of retrieval performance compared to existing privacy-preserving methods.

Problem

• Protecting text embeddings from Embedding Inversion Attacks (EIAs).
• Maintaining retrieval accuracy in end-cloud collaboration systems.
• Enhancing privacy without modifying the embedding model.

Innovation

• Entropy-driven perturbation for privacy protection
• Bounded perturbations maintain retrieval accuracy
• Direct integration without embedding model changes