LHDNN: Maintaining High Precision and Low Latency Inference of Deep Neural Networks on Encrypted Data
Source abstract
Abstract
The advancement of deep neural networks (DNNs) has prompted many cloud service providers to offer deep learning as a service (DLaaS) to users across various application domains. However, in current DLaaS prediction systems, users’ data are at risk of leakage. Homomorphic encryption allows operations to be performed on ciphertext without decryption, which can be applied to DLaaS to ensure users’ data privacy. However, mainstream homomorphic encryption schemes only support homomorphic addition and multiplication, and do not support the ReLU activation function commonly used in the activation layers of DNNs. Previous work used approximate polynomials to replace the ReLU activation function, but the DNNs they implemented either had low inference accuracy or high inference latency. In order to achieve low inference latency of DNNs on encrypted data while ensuring inference accuracy, we propose a low-degree Hermite deep neural network framework (called LHDNN), which uses a set of low-degree trainable Hermite polynomials (called LotHps) as activation layers of DNNs. Additionally, LHDNN integrates a novel weight initialization and regularization module into the LotHps activation layer, which makes the training process of DNNs more stable and gives a stronger generalization ability. Additionally, to further improve the model accuracy, we propose a variable-weighted difference training (VDT) strategy that uses ReLU-based models to guide the training of LotHps-based models. Extensive experiments on multiple benchmark datasets validate the superiority of LHDNN in terms of inference speed and accuracy on encrypted data.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Cryptography and Data Security · Privacy-Preserving Technologies in Data · Cryptography and Residue Arithmetic
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mathematics and Statistics, Henan University of Science and Technology, Luoyang 471023, China
Intelligent System Science and Technology Innovation Center, Longmen Laboratory, Luoyang 471023, China
Sports Big Data Center, Department of Physical Education, Zhengzhou University, Zhengzhou 450001, China
Funding
National Natural Science Foundation of China · Colleges and Universities of Henan Province of China · Key Science and Technology Project of Henan Province of China
References (39)
Show source-provided references
1. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions10.1186/s40537-021-00444-8 ↗
2. Securely outsourcing neural network inference to the cloud with lightweight techniques10.1109/TDSC.2022.3141391 ↗
3. A review of privacy-preserving techniques for deep learning10.1016/j.neucom.2019.11.041 ↗
4. Gentry, C. (June, January 31). Fully homomorphic encryption using ideal lattices. Proceedings of the Forty-First Annual ACM Symposium on Theory of Computing, Bethesda, MD, USA.
5. Privacy-preserving deep learning with homomorphic encryption: An introduction10.1109/MCI.2022.3180883 ↗
6. Optimizing Privacy-Preserving Outsourced Convolutional Neural Network Predictions10.1109/TDSC.2020.3029899 ↗
7. Wang, J., He, D., Castiglione, A., Gupta, B.B., Karuppiah, M., and Wu, L. (2022). Pcnncec: Efficient and privacy-preserving convolutional neural network inference based on cloud-edge-client collaboration. IEEE Trans. Netw. Sci. Eng.10.1109/TNSE.2022.3177755 ↗
8. SecureTrain: An Approximation-Free and Computationally Efficient Framework for Privacy-Preserved Neural Network Training10.1109/TNSE.2020.3040704 ↗
9. Dowlin, N., Gilad-Bachrach, R., Laine, K., Lauter, K., Naehrig, M., and Wernsing, J. (2016, January 19–24). CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA.
10. Privacy-preserving classification on deep neural network
11. Mohassel, P., and Zhang, Y. (2017, January 22–26). Secureml: A system for scalable privacy-preserving machine learning. Proceedings of the 2017 IEEE Symposium on Security And Privacy (SP), San Jose, CA, USA.10.1109/SP.2017.12 ↗
12. Hesamifard, E., Takabi, H., and Ghasemi, M. (2019, January 25–27). Deep Neural Networks Classification over Encrypted Data. Proceedings of the 9th ACM Conference on Data and Application Security and Privacy (CODASPY), Richardson, TX, USA.10.1145/3292006.3300044 ↗
13. Lee, J., Lee, E., Lee, J.-W., Kim, Y., Kim, Y.-S., and No, J.-S. (2021). Precise approximation of convolutional neural networks for homomorphic ally encrypted data. arXiv.
14. Privacy-Preserving Machine Learning With Fully Homomorphic Encryption for Deep Neural Network10.1109/ACCESS.2022.3159694 ↗
15. Bourse, F., Minelli, M., Minihold, M., and Paillier, P. (2018, January 19–23). Fast homomorphic evaluation of deep discretized neural networks. Proceedings of the Annual International Cryptology Conference, Santa Barbara, CA, USA.10.1007/978-3-319-96878-0_17 ↗
16. Sanyal, A., Kusner, M., Gascon, A., and Kanade, V. (2018, January 10–15). TAPAS: Tricks to accelerate (encrypted) prediction as a service. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
17. Lou, Q., and Jiang, L. (2019, January 8–14). SHE: A Fast and Accurate Deep Neural Network for Encrypted Data. Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
18. REDsec: Running Encrypted DNNs in Seconds
19. DOReN: Toward Efficient Deep Convolutional Neural Networks with Fully Homomorphic Encryption10.1109/TIFS.2021.3090959 ↗
20. Towards high performance homomorphic encryption for inference tasks on CPU: An MPI approach10.1016/j.future.2022.03.033 ↗
21. Alsaedi, E.M., Farhan, A.K., Falah, M.W., and Oleiwi, B.K. (2022, January 29–31). Classification of Encrypted Data Using Deep Learning and Legendre Polynomials. Proceedings of the ICR’22 International Conference on Innovations in Computing Research, Athens, Greece.10.1007/978-3-031-14054-9_31 ↗
22. Yagyu, K., Takeuchi, R., Nishigaki, M., and Ohki, T. (2022, January 27–29). Improving Classification Accuracy by Optimizing Activation Function for Convolutional Neural Network on Homomorphic Encryption. Advances on Broad-Band Wireless Computing, Communication and Applications. Proceedings of the 17th International Conference on Broad-Band Wireless Computing, Communication and Applications (BWCCA-2022), Tirana, Albania.10.1007/978-3-031-20029-8_10 ↗
23. Cheon, J.H., Kim, A., Kim, M., and Song, Y. (2017, January 3–7). Homomorphic encryption for arithmetic of approximate numbers. Proceedings of the International Conference on the Theory and Application of Cryptology and Information Security, Hong Kong, China.10.1007/978-3-319-70694-8_15 ↗
24. Cheon, J.H., Han, K., Kim, A., Kim, M., and Song, Y. (2018, January 15–17). A full RNS variant of approximate homomorphic encryption. Proceedings of the International Conference on Selected Areas in Cryptography, Calgary, AB, Canada.10.1007/978-3-030-10970-7_16 ↗
25. Cheon, J.H., Han, K., Kim, A., Kim, M., and Song, Y. (May, January 29). Bootstrapping for approximate homomorphic encryption. Proceedings of the Annual International Conference on the Theory and Applications of Cryptographic Techniques, Tel Aviv, Israel.
26. Constructive feedforward neural networks using hermite polynomial activation functions10.1109/TNN.2005.851786 ↗
27. Bottou, L. (2012). Neural Networks: Tricks of the Trade, Springer.
28. Baruch, M., Drucker, N., Greenberg, L., and Moshkowich, G. (2022, January 20–23). A methodology for training homomorphic encryption friendly neural networks. Proceedings of the International Conference on Applied Cryptography and Network Security, Rome, Italy.10.1007/978-3-031-16815-4_29 ↗
29. LeCun, Y., Cortes, C., and Burges, C. (2022, October 18). The MNIST Database of Handwritten Digits. Available online: http://yann.lecun.com/exdb/mnist/.
30. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions10.1038/sdata.2018.161 ↗
31. Krizhevsky, A., Nair, V., and Hinton, G. (2010). CIFAR-10, Canadian Institute for Advanced Research.
32. ImageNet Classification with Deep Convolutional Neural Networks10.1145/3065386 ↗
33. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
34. Aharoni, E., Adir, A., Baruch, M., Drucker, N., Ezov, G., Farkash, A., Greenberg, L., Masalha, R., Moshkowich, G., and Murik, D. (2020). HeLayers: A Tile Tensors Framework for Large Neural Networks on Encrypted Data. arXiv.
35. Garimella, K., Jha, N.K., and Reagen, B. (2021). Sisyphus: A cautionary tale of using low-degree polynomial activations in privacy-preserving deep learning. arXiv.
36. HE-Booster: An Efficient Polynomial Arithmetic Acceleration on GPUs for Fully Homomorphic Encryption10.1109/TPDS.2022.3228628 ↗
37. Chou, E., Beal, J., Levy, D., Yeung, S., Haque, A., and Fei-Fei, L. (2018). Faster cryptonets: Leveraging sparsity for real-world encrypted inference. arXiv.
38. Lee, E., Lee, J.-W., Lee, J., Kim, Y.-S., Kim, Y., No, J.-S., and Choi, W. (2022, January 17–23). Low-complexity deep convolutional neural networks on fully homomorphic encryption using multiplexed parallel convolutions. Proceedings of the International Conference on Machine Learning, Baltimore, MA, USA.
39. Towards the alexnet moment for homomorphic encryption: Hcnn, the first homomorphic cnn on encrypted data with gpus