Privacy-Preserving Deep Learning Framework Based on Restricted Boltzmann Machines and Instance Reduction Algorithms
Source abstract
Abstract
The combination of collaborative deep learning and Cyber-Physical Systems (CPSs) has the potential to improve decision-making, adaptability, and efficiency in dynamic and distributed environments. However, it brings privacy, communication, and resource restrictions concerns that must be properly addressed for successful implementation in real-world CPS systems. Various privacy-preserving techniques have been proposed, but they often add complexity and decrease accuracy and utility. In this paper, we propose a privacy-preserving deep learning framework that combines Instance Reduction Techniques (IR) and the Restricted Boltzmann Machine (RBM) to preserve privacy while overcoming the limitations of other frameworks. The RBM encodes training data to retain relevant features, and IR selects the relevant encoded instances to send to the server for training. Privacy is preserved because only a small subset of the training data is sent to the server. Moreover, it is sent after encoding it using RBM. Experiments show that our framework preserves privacy with little loss of accuracy and a substantial reduction in training time. For example, using our framework, a CNN model for the MNIST dataset achieves 96% accuracy compared to 99% in a standard collaborative framework (with no privacy measures taken), with training time reduced from 133.259 s to 99.391 s. Our MLP model for MNIST achieves 97% accuracy compared to 98% in the standard collaborative framework, with training time reduced from 118.146 s to 87.873 s. Compared to other studies, our method is a simple approach that protects privacy, maintains the utility of deep learning models, and reduces training time and communication costs.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Adversarial Robustness in Machine Learning · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, College of Computer & Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia
Funding
Deputyship for Research and Innovation, “Ministry of Education” in Saudi Arabia
References (76)
Show source-provided references
1. Shokri, R., and Shmatikov, V. (October, January 29). Privacy-Preserving Deep Learning. Proceedings of the 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA.10.1109/ALLERTON.2015.7447103 ↗
2. Fredrikson, M., Jha, S., and Ristenpart, T. (2015, January 12–16). Model Inversion Attacks That Exploit Confidence Information and Basic Countermeasures. Proceedings of the ACM Conference on Computer and Communications Security, Denver, CO, USA.10.1145/2810103.2813677 ↗
3. Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., and Erlingsson, Ú. (2021, January 11–13). Extracting Training Data from Large Language Models. Proceedings of the 30th USENIX Security Symposium, Vancouver, BC, Canada.
4. Wang, B., and Gong, N.Z. (2018, January 21–23). Stealing Hyperparameters in Machine Learning. Proceedings of the IEEE Symposium on Security and Privacy, San Francisco, CA, USA.10.1109/SP.2018.00038 ↗
5. Tramèr, F., Zhang, F., Juels, A., Reiter, M.K., and Ristenpart, T. (2016, January 10–12). Stealing Machine Learning Models via Prediction APIs. Proceedings of the 25th USENIX Security Symposium, Austin, TX, USA.
6. Zhang, D., Chen, X., Wang, D., and Shi, J. (2018, January 18–21). A Survey on Collaborative Deep Learning and Privacy-Preserving. Proceedings of the 2018 IEEE 3rd International Conference on Data Science in Cyberspace, (DSC 2018), Guangzhou, China.10.1109/DSC.2018.00104 ↗
7. Keerthi, C.K., Jabbar, M.A., and Seetharamulu, B. (2017, January 14–16). Cyber Physical Systems(CPS):Security Issues, Challenges and Solutions. Proceedings of the 2017 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), Coimbatore, India.10.1109/ICCIC.2017.8524312 ↗
8. A Review of Deep Learning Security and Privacy Defensive Techniques
9. Privacy-Preserving Deep Learning on Machine Learning as a Service-a Comprehensive Survey10.1109/ACCESS.2020.3023084 ↗
10. Brendan McMahan, H., Moore, E., Ramage, D., Hampson, S., and Agüera y Arcas, B. (2017, January 20–22). Communication-Efficient Learning of Deep Networks from Decentralized Data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, Fort Lauderdale, FL, USA.
11. Xie, P., Bilenko, M., Finley, T., Gilad-Bachrach, R., Lauter, K., and Naehrig, M. (2014). Crypto-Nets: Neural Networks over Encrypted Data. arXiv.
12. Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H.B., Patel, S., Ramage, D., Segal, A., and Seth, K. (November, January 30). Practical Secure Aggregation for Privacy-Preserving Machine Learning. Proceedings of the ACM Conference on Computer and Communications Security, Dallas, TX, USA.
13. Schunter, M. (2016, January 24–28). Intel Software Guard Extensions. Proceedings of the 2016 ACM Workshop on Software Protection, Vienna, Austria.10.1145/2995306.2995307 ↗
14. Reduction Techniques for Instance-Based Learning Algorithms10.1023/A:1007626913721 ↗
15. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
16. Gradient-Based Learning Applied to Document Recognition10.1109/5.726791 ↗
17. Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv.
18. Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. (2017, January 14–19). EMNIST: Extending MNIST to Handwritten Letters. Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA.10.1109/IJCNN.2017.7966217 ↗
19. Hinton, G.E. (2012). A Practical Guide to Training Restricted Boltzmann Machines, Springer.10.1007/978-3-642-35289-8_32 ↗
20. A Comprehensive Survey and Analysis of Generative Models in Machine Learning10.1016/j.cosrev.2020.100285 ↗
21. Enhancing Anomaly Detection through Restricted Boltzmann Machine Features Projection
22. Ghojogh, B., Ghodsi, A., Karray, F., and Crowley, M. (2021). Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey. arXiv.10.1007/978-3-031-10602-6_18 ↗
23. Kondratyev, A., Schwarz, C., and Horvath, B. (2020). Data Anonymisation, Outlier Detection and Fighting Overfitting with Restricted Boltzmann Machines. SSRN Electron. J., 1–27.10.2139/ssrn.3526436 ↗
24. Bishop, C., and Ligne, S. (2006). Pattern Recognition and Machine Learning, Springer.
25. Aggarwal, C.C. (2018). Neural Networks and Deep Learning, Springer International Publishing AG.10.1007/978-3-319-94463-0 ↗
26. Witten, I.H., Frank, E., and Hall, M. (2005). Data Mining: Practical Machine Learning Tools and Techniques, Elsevier.
27. Smoothing Decision Boundaries to Avoid Overfitting in Neural10.14311/NNW.2011.21.019 ↗
28. El Hindi, K., and Alakhras, M. Proceedings of the IADIS Multi Conference on Computer Science of Intelligent Systems and Agents 2009, Algarve, Portugal, 21–23 June 2009.
29. Data Selection for Neural Networks
30. A Review of Instance Selection Methods10.1007/s10462-010-9165-y ↗
31. Carbonera, J.L., and Abel, M. (2016, January 6–8). A Novel Density-Based Approach for Instance Selection. Proceedings of the 2016 IEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI), San Jose, CA, USA.10.1109/ICTAI.2016.0090 ↗
32. Grochowski, M., and Jankowski, N. (2004, January 7–11). Comparison of Instance Selection Algorithms II. Results and Comments. Proceedings of the International Conference on Artificial Intelligence and Soft Computing, Zakopane, Poland.10.1007/978-3-540-24844-6_87 ↗
33. A New Approach for Instance Selection: Algorithms, Evaluation, and Comparisons10.1016/j.eswa.2020.113297 ↗
34. Asymptotic Properties of Nearest Neighbor Rules Using Edited Data10.1109/TSMC.1972.4309137 ↗
35. Nasr, M., Shokri, R., and Houmansadr, A. (2019, January 19–23). Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-Box Inference Attacks against Centralized and Federated Learning. Proceedings of the IEEE Symposium on Security and Privacy, San Francisco, CA, USA.10.1109/SP.2019.00065 ↗
36. A Survey on Deep Learning Techniques for Privacy-Preserving
37. The Algorithmic Foundations of Differential Privacy10.1561/0400000042 ↗
38. Abadi, M., McMahan, H.B., Chu, A., Mironov, I., Zhang, L., Goodfellow, I., and Talwar, K. (2016, January 24–28). Deep Learning with Differential Privacy. Proceedings of the ACM Conference on Computer and Communications Security, Vienna, Austria.10.1145/2976749.2978318 ↗
39. Preserving Differential Privacy in Convolutional Deep Belief Networks10.1007/s10994-017-5656-2 ↗
40. Local Differential Privacy for Deep Learning10.1109/JIOT.2019.2952146 ↗
41. Chen, Z., and Wang, Y. (2023). Locally Differentially Private Distributed Online Learning with Guaranteed Optimality. arXiv.
42. Jayaraman, B., and Evans, D. (2019, January 14–16). Evaluating Differentially Private Machine Learning in Practice. Proceedings of the 28th USENIX Security Symposium, Santa Clara, CA, USA.
43. Truex, S., Steinke, T., Baracaldo, N., Ludwig, H., Zhou, Y., Anwar, A., and Zhang, R. (2019, January 15). A Hybrid Approach to Privacy-Preserving Federated Learning. Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, London, UK.10.1145/3338501.3357370 ↗
44. Federated Learning with Differential Privacy: Algorithms and Performance Analysis10.1109/TIFS.2020.2988575 ↗
45. Efficient and Privacy-Preserving Group Signature for Federated Learning10.1016/j.future.2023.04.017 ↗
46. Advances and Open Problems in Federated Learning10.1561/2200000083 ↗
47. On Data Banks and Privacy Homomorphisms
48. Hesamifard, E., Takabi, H., and Ghasemi, M. (2017). CryptoDL: Deep Neural Networks over Encrypted Data. arXiv.
49. Privacy-Preserving Deep Learning via Additively Homomorphic Encryption10.1109/TIFS.2017.2787987 ↗
50. Privacy-Preserving Deep Learning via Weight Transmission10.1109/TIFS.2019.2911169 ↗
51. Towards the AlexNet Moment for Homomorphic Encryption: HCNN, the First Homomorphic CNN on Encrypted Data with GPUs10.1109/TETC.2020.3014636 ↗
52. Privacy-Preserving Neural Networks with Homomorphic Encryption: Challenges and Opportunities10.1007/s12083-021-01076-8 ↗
53. Secure Multi-Party Computation
54. Rouhani, B.D., Riazi, M.S., and Koushanfar, F. (2018, January 24–29). Deepsecure: Scalable Provably-Secure Deep Learning. Proceedings of the 55th Annual Design Automation Conference, San Francisco, CA, USA.10.1145/3195970.3196023 ↗
55. SecureNN: 3-Party Secure Computation for Neural Network Training10.2478/popets-2019-0035 ↗
56. Kumar, N., Rathee, M., Chandran, N., Gupta, D., Rastogi, A., and Sharma, R. (2020, January 18–21). CrypTFlow: Secure TensorFlow Inference. Proceedings of the IEEE Symposium on Security and Privacy, San Francisco, CA, USA.10.1109/SP40000.2020.00092 ↗
57. Tan, S., Knott, B., Tian, Y., and Wu, D.J. (2021, January 24–27). CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU. Proceedings of the IEEE Symposium on Security and Privacy, San Francisco, CA, USA.10.1109/SP40001.2021.00098 ↗
58. CrypTen: Secure Multi-Party Computation Meets Machine Learning
59. Tramèr, F., Shokri, R., Joaquin, A.S., Le, H., Jagielski, M., Hong, S., and Carlini, N. (2022). Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets. arXiv.10.1145/3548606.3560554 ↗
60. DP-Cryptography: Marrying Differential Privacy and Cryptography in Emerging Applications10.1145/3418290 ↗
61. Hunt, T., Song, C., Shokri, R., Shmatikov, V., and Witchel, E. (2018). Chiron: Privacy-Preserving Machine Learning as a Service. arXiv.
62. Tramèr, F., and Boneh, D. (2019, January 6–9). Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware. Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA.
63. Abay, N., Zhou, Y., Kantarcioglu, M., Thuraisingham, B., and Sweeney, L. (2018, January 10–14). Privacy Preserving Synthetic Data Release Using Deep Learning. Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Dublin, Ireland.10.1007/978-3-030-10925-7_31 ↗
64. Huang, Y., Song, Z., Li, K., and Arora, S. (2020, January 13–18). InstaHide: Instance-Hiding Schemes for Private Distributed Learning. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, Virtual Event.
65. Carlini, N., Deng, S., Garg, S., Jha, S., Mahloujifar, S., Mahmoody, M., Thakurta, A., and Tramer, F. (2021, January 24–27). Is Private Learning Possible with Instance Encoding?. Proceedings of the IEEE Symposium on Security and Privacy, San Francisco, CA, USA.10.1109/SP40001.2021.00099 ↗
66. Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. (2018, January 9–12). Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. Proceedings of the IEEE Computer Security Foundations Symposium, Oxford, UK.10.1109/CSF.2018.00027 ↗
67. Roder, M., de Rosa, G.H., and Papa, J.P. (2020). Learnergy: Energy-Based Machine Learners. arXiv.
68. (2023, January 03). Colaboratory. Available online: https://colab.research.google.com/.
69. Imbalanced-Learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning
70. Automatic Differentiation in PCF10.1145/3434309 ↗
71. Individual Comparisons by Ranking Methods10.2307/3001968 ↗
72. Mohassel, P., and Zhang, Y. SecureML: A System for Scalable Privacy-Preserving Machine Learning. Proceedings of the IEEE Symposium on Security and Privacy.
73. Liu, J., Juuti, M., Lu, Y., and Asokan, N. (November, January 30). Oblivious Neural Network Predictions via MiniONN Transformations. Proceedings of the ACM Conference on Computer and Communications Security, Dallas, TX, USA.
74. Sadegh Riazi, M., Songhori, E.M., Weinert, C., Schneider, T., Tkachenko, O., and Koushanfar, F. (2018, January 4–8). Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applications. Proceedings of the ASIACCS 2018—2018 ACM Asia Conference on Computer and Communications Security, Incheon, Republic of Korea.10.1145/3196494.3196522 ↗
75. Mohassel, P., and Rindal, P. (2018, January 15–19). ABY3: A Mixed Protocol Framework for Machine Learning. Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security—CCS ’18, Toronto, ON, Canada.
76. Dowlin, N., Gilad-Bachrach, R., Laine, K., Lauter, K., Naehrig, M., and Wernsing, J. (2016, January 19–24). CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy. Proceedings of the International Conference on Machine Learning, New York, NY, USA.