Centaur: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inference

2024-12-14

arXiv.org

AI Summary

Addressing the “impossibility triangle” in privacy-preserving Transformer inference (PPTI)—where privacy, efficiency, and model accuracy are mutually conflicting—this paper proposes Centaur, a hybrid framework. Centaur introduces a novel heterogeneous dual-path protection mechanism: random parameter permutation for model weights and secure multi-party computation (SMPC) for inference inputs, synergistically integrated with Transformer-structure-aware algorithms and efficient encrypted tensor operations. It is the first framework to jointly optimize privacy guarantees, inference latency, and prediction accuracy within a unified architecture. Experiments demonstrate robust resistance against diverse model inversion attacks; inference accuracy matches plaintext performance exactly; and compared to end-to-end SMPC baselines, Centaur achieves 5.0×–30.4× speedup while preserving strong privacy.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

As pre-trained models, like Transformers, are increasingly deployed on cloud platforms for inference services, the privacy concerns surrounding model parameters and inference data are becoming more acute. Current Privacy-Preserving Transformer Inference (PPTI) frameworks struggle with the"impossible trinity"of privacy, efficiency, and performance. For instance, Secure Multi-Party Computation (SMPC)-based solutions offer strong privacy guarantees but come with significant inference overhead and performance trade-offs. On the other hand, PPTI frameworks that use random permutations achieve inference efficiency close to that of plaintext and maintain accurate results but require exposing some model parameters and intermediate results, thereby risking substantial privacy breaches. Addressing this"impossible trinity"with a single technique proves challenging. To overcome this challenge, we propose Centaur, a novel hybrid PPTI framework. Unlike existing methods, Centaur protects model parameters with random permutations and inference data with SMPC, leveraging the structure of Transformer models. By designing a series of efficient privacy-preserving algorithms, Centaur leverages the strengths of both techniques to achieve a better balance between privacy, efficiency, and performance in PPTI. We comprehensively evaluate the effectiveness of Centaur on various types of Transformer models and datasets. Experimental results demonstrate that the privacy protection capabilities offered by Centaur can withstand various existing model inversion attack methods. In terms of performance and efficiency, Centaur not only maintains the same performance as plaintext inference but also improves inference speed by $5.0-30.4$ times.

Problem

• Balancing privacy, efficiency, and performance in Privacy-Preserving Transformer Inference
• Overcoming high computational overhead in Secure Multi-Party Computation approaches
• Addressing privacy compromises in permutation-based methods for PPTI

Innovation

• Integrates random permutations and SMPC
• Tailors PPTI algorithms for Transformers
• Boosts speed 5.0-30.4 times