Privacy-Preserving CNN Training with Transfer Learning

2023-04-07

arXiv.org

AI Summary

This work addresses the low efficiency and poor accuracy of deep learning model training under homomorphic encryption (HE). We propose the first end-to-end CNN training framework operating entirely in the HE domain. Leveraging transfer learning via RegNet_X_400MF, we reduce the task to multi-class logistic regression within the encrypted space. Key innovations include: (1) a quadratic-gradient optimization method to accelerate convergence; (2) a differentiable sigmoid-based approximation of softmax; (3) a squared-likelihood error loss function; and (4) the Volley Revolver matrix encoding scheme to enhance ciphertext computational density. Evaluated on a 64-core cloud server, our framework completes two training iterations in just 21 minutes, reduces uploaded ciphertexts to six, and achieves 21.49% test accuracy on an MNIST subset. All code is fully open-sourced.

Abstract

In this paper, we present a practical solution to implement privacy-preserving CNN training based on mere Homomorphic Encryption (HE) technique. To our best knowledge, this is the first attempt successfully to crack this nut and no work ever before has achieved this goal. Several techniques combine to accomplish the task:: (1) with transfer learning, privacy-preserving CNN training can be reduced to homomorphic neural network training, or even multiclass logistic regression (MLR) training; (2) via a faster gradient variant called $ exttt{Quadratic Gradient}$, an enhanced gradient method for MLR with a state-of-the-art performance in convergence speed is applied in this work to achieve high performance; (3) we employ the thought of transformation in mathematics to transform approximating Softmax function in the encryption domain to the approximation of the Sigmoid function. A new type of loss function termed $ exttt{Squared Likelihood Error}$ has been developed alongside to align with this change.; and (4) we use a simple but flexible matrix-encoding method named $ exttt{Volley Revolver}$ to manage the data flow in the ciphertexts, which is the key factor to complete the whole homomorphic CNN training. The complete, runnable C++ code to implement our work can be found at: href{https://github.com/petitioner/HE.CNNtraining}{$ exttt{https://github.com/petitioner/HE.CNNtraining}$}. We select $ exttt{REGNET_X_400MF}$ as our pre-trained model for transfer learning. We use the first 128 MNIST training images as training data and the whole MNIST testing dataset as the testing data. The client only needs to upload 6 ciphertexts to the cloud and it takes $sim 21$ mins to perform 2 iterations on a cloud with 64 vCPUs, resulting in a precision of $21.49%$.

Problem

• Privacy Protection
• Transfer Learning
• Homomorphic Encryption

Innovation

• Homomorphic Encryption
• Transfer Learning
• Privacy-Preserving CNN Training