Privacy Auditing in Differential Private Machine Learning: The Current Trends
Source abstract
Abstract
Differential privacy has recently gained prominence, especially in the context of private machine learning. While the definition of differential privacy makes it possible to provably limit the amount of information leaked by an algorithm, practical implementations of differentially private algorithms often contain subtle vulnerabilities. Therefore, there is a need for effective methods that can audit (ϵ,δ) differentially private algorithms before they are deployed in the real world. The article examines studies that recommend privacy guarantees for differential private machine learning. It covers a wide range of topics on the subject and provides comprehensive guidance for privacy auditing schemes based on privacy attacks to protect machine-learning models from privacy leakage. Our results contribute to the growing literature on differential privacy in the realm of privacy auditing and beyond and pave the way for future research in the field of privacy-preserving models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Cryptography and Data Security · Blockchain Technology Applications and Security
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Electronics and Computer Science, 14 Dzerbenes St., LV-1006 Riga, Latvia
Funding
Chips Joint Undertaking
References (100+)
Show source-provided references
1. Choudhury, O., Gkoulalas-Divanis, A., Salonidis, T., Sylla, I., Park, Y., Hsu, G., and Das, A. (2019). Differential Privacy-Enabled Federated Learning For Sensitive Health Data. arXiv, Available online: https://arxiv.org/abs/1910.02578.
2. Halevi, S., and Rabin, T. (2006). Calibrating Noise To Sensitivity In Private Data Analysis. Theory of Cryptography, Springer.
3. Williamson, S.M., and Prybutok, V. (2024). Balancing Privacy and Progress: A Review of Privacy Challenges, Systemic Oversight, and Patient Perceptions. In: AI-Driven Healthcare. Appl. Sci., 14.10.3390/app14020675 ↗
4. Barbierato, E., and Gatti, A. (2024). The Challenges of Machine Learning: A Critical Review. Electronics, 13.10.3390/electronics13020416 ↗
5. Noor, M.H.M., and Ige, A.O. (2024). A Survey on State-of-the-art Deep Learning Applications and Challenges. arXiv, Available online: https://arxiv.org/abs/2403.17561.
6. Du Pin Calmon, F., and Fawaz, N. (2012, January 1–5). Privacy Against Statistical Inference. Proceedings of the 2012 50th Annual Allerton Conference on Communication, Control, and Computing, Allerton, Monticello, IL, USA.10.1109/Allerton.2012.6483382 ↗
7. Dehghani, M., Azarbonyad, H., Kamps, J., and de Rijke, M. (2017). Share your Model of your Data: Privacy Preserving Mimic Learning for Mimic Learning for Ranking. arXiv, Available online: https://arxiv.org/abs/1707.07605.
8. An Empirical Study Of Pattern Leakage Impact During Data Preprocessing on Machine Learning-Based Intrusion Detection Models Reliability10.1016/j.eswa.2023.120715 ↗
9. Machine Unlearning: Solutions and Challenges10.1109/TETCI.2024.3379240 ↗
10. Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. (2018, January 9–12). Privacy risk in machine learning: Analyzing the connection to overfitting. Proceedings of the 2018 IEEE 31st Computer Security Foundations Symposium (CSF), Oxford, UK.10.1109/CSF.2018.00027 ↗
11. Model Architecture Level Privacy Leakage In Neural Networks
12. Bounding information leakage in machine learning10.1016/j.neucom.2023.02.058 ↗
13. McSherry, F., and Talwar, K. (2007, January 20–23). Mechanism Design via Differential Privacy. Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2007), Providence, RI, USA.10.1109/FOCS.2007.4389483 ↗
14. Mulder, V., and Humbert, M. (2023). Differential privacy. Trends in Data Protection and Encryption Technologies, Springer.10.1007/978-3-031-33386-6 ↗
15. A Survey on Differential Private Machine Learning10.1109/MCI.2020.2976185 ↗
16. When Machine Learning Meets Privacy: A Survey and Outlook
17. Critical Review on the Use (and Misuse) of Differential Privacy in Machine Learning
18. Protecting Decision Boundary of Machine Learning Model With Differential Private Perturbation10.1109/TDSC.2020.3043382 ↗
19. A Practical Guide to Machine Learning with Differential Privacy10.1613/jair.1.14649 ↗
20. Choquette-Choo, C.A., Dullerud, N., Dziedzic, A., Zhang, Y., Jha, S., Papernot, N., and Wang, X. (2021, January 4). CaPC Learning: Confidential and Private Collaborative Learning. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria.
21. Jayaraman, B., and Evans, D. (2019, January 14–16). Evaluating Differentially Private Machine Learning in Practice. Proceedings of the 28th USENIX Security Symposium (USENIX Security 19), Santa Clara, CA, USA.
22. Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., and Jégou, H. (2019, January 9–15). White-box vs black-box: Bayes optimal strategies for membership inference. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA.
23. Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., and Zhang, L. (2016, January 24–28). Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS), Vienna, Austria.10.1145/2976749.2978318 ↗
24. Differential Privacy Has Disparate Impact on Model Accuracy
25. Differential Private Empirical Risk Minimization under the Fairness Lens
26. Bichsel, B., Stefen, S., Bogunovic, I., and Vechev, M. (2021, January 24–27). Dp-Sniper: Black-Box Discovery Of Differential Privacy Violations Using Classifiers. Proceedings of the 2021 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA.10.1109/SP40001.2021.00081 ↗
27. Bugliesi, M., Preneel, B., Sassone, V., and Wegener, I. (2006). Differential Privacy. Automata, Languages and Programming, Springer. Lecture Notes in Computer Science.10.1007/11787006 ↗
28. Differential Private Noise Adding Mechanism and Its Application on Consensus Algorithm10.1109/TSP.2020.3006760 ↗
29. Differentially Private Data Publishing for Arbitrary Partitioned Data10.1016/j.ins.2020.10.051 ↗
30. Baraheem, S.S., and Yao, Z. (2022). A Survey on Differential Privacy with Machine Learning and Future Outlook. arXiv, Available online: https://arxiv.org/abs/2211.10708.
31. The Algorithmic Foundations Of Differential Privacy10.1561/0400000042 ↗
32. Chadha, K., Jagielski, M., Papernot, N., Choquette-Choo, C.A., and Nasr, M. (2024). Auditing Private Prediction. arXiv.
33. Papernot, N., Abadi, M., Erlingsson, Ú., Goodfelow, I., and Talwar, K. (2024, December 01). Semi-Supervise Knowledge Transfer for Deep Learning from Private Training Data. International Conference on Learning Representations. Available online: https://openreview.net/forum?id=HkwoSDPgg.
34. Bernau, D., Robl, J., Grassal, P.W., Schneider, S., and Kerschbaum, F. (2021). Comparing Local and Central Differential Privacy Using Membership Inference Attacks. IFIP Annual Conference on Data and Applications Security and Privacy, Springer.10.1007/978-3-030-81242-3_2 ↗
35. Hsu, J., Gaboardi, M., Haeberlen, A., Khanna, S., Narayan, A., Pierce, B.C., and Roth, A. (2014, January 19–22). Differential Privacy: An economic methods for choosing epsilon. Proceedings of the Computer Security Foundations Workshop, Vienna, Austria.10.1109/CSF.2014.35 ↗
36. Mehner, L., Voigt, S.N.V., and Tschorsch, F. (2021, January 6–10). Towards Explaining Epsilon: A Worst-Case Study of Differential Privacy Risks. Proceedings of the 2021 IEEE European Symposium on Security and Privacy Workshop, Euro S and PW, Virtual.10.1109/EuroSPW54576.2021.00041 ↗
37. Busa-Fekete, R.I., Dick, T., Gentile, C., Medina, A.M., Smith, A., and Swanberg, M. (2024). Auditing Privacy Mechanisms via Label Inference Attacks. arXiv, Available online: https://arxiv.org/abs/2406.02797.
38. Desfontaines, D., and Pejó, B. (2022). SoK: Differential Privacies. arXiv, Available online: https://arxiv.org/abs/1906.01337.
39. Lycklama, H., Viand, A., Küchler, N., Knabenhans, C., and Hithnawi, A. (2024). Holding Secrets Accountable: Auditing Privacy-Preserving Machine Learning. arXiv, Available online: https://arxiv.org/abs/2402.15780.
40. Kong, W., Medina, A.M., Ribero, M., and Syed, U. (2023). DP-Auditorium: A Large Scale Library for Auditing Differential Privacy. arXiv, Available online: https://arxiv.org/abs/2307.05608.
41. Differential Privacy Under Membership Inference Attacks
42. Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramer, F., Jagielski, M., Carlini, N., and Terzis, A. (2023, January 9–11). Tight Auditing of Differentially Private Machine Learning. Proceedings of the 32nd USENIX Security Symposium (USENIX Security 23), San Francisco, CA, USA.
43. Steinke, T., Nasr, M., and Jagielski, M. (2023). Privacy Auditing with One (1) Training Run. arXiv, Available online: https://arxiv.org/abs/2305.08846.
44. Assessment and Privacy Risk Assessment Methodologies: A Systematic Literature Review10.1109/ACCESS.2024.3360864 ↗
45. Ding, Z., Wang, Y., Wang, G., Zhang, D., and Kifer, D. (2018, January 15–19). Detecting Violations Of Differential Privacy. Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, Toronto, ON, Canada.10.1145/3243734.3243818 ↗
46. Auditing Differentially Private Machine Learning: How Private is Private sgd?
47. Nasr, M., Songi, S., Thakurta, A., Papernot, N., and Carlin, N. (2021, January 24–27). Adversary instantiation: Lower bounds for differentially private machine learning. Proceedings of the 2021 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA.10.1109/SP40001.2021.00069 ↗
48. Lokna, J., Paradis, A., Dimitrov, D.I., and Vechev, M. (2023, January 26–30). Group and Attack: Auditing Differential Privacy. Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS ’23), Copenhagen, Denmark. Available online: https://dl.acm.org/doi/10.1145/3576915.3616607.10.1145/3576915.3616607 ↗
49. Tramèr, F., Terzis, A., Steinke, T., Song, S., Jagielski, M., and Carlini, N. (2022). Debugging differential privacy: A case study for privacy auditing. arXiv, Available online: https://arxiv.org/abs/2202.12219.
50. Kifer, D., Messing, S., Roth, A., Thakurta, A., and Zhang, D. (2020). Guidelines for Implementing and Auditing Differentially Private Systems. arXiv, Available online: https://arxiv.org/abs/2002.04049.
51. Homer, N., Szelinger, S., Redman, M., Duggan, D., Tembe, W., Muehling, J., Pearson, J.V., Stephan, D.A., Nelson, S.F., and Craig, D.W. (2008). Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS Genet., 4.10.1371/journal.pgen.1000167 ↗
52. Shokri, R., Stronati, M., Song, C., and Shmatikov, V. (2017, January 22–26). Membership Inference Attacks against Machine Learning Models. Proceedings of the 2017 IEEE Symposium on Security and Priavcy (S&P), San Jose, CA, USA.10.1109/SP.2017.41 ↗
53. Cui, G., Ge, L., Zhao, Y., and Fang, T. (2024, January 9–11). A Membership Inference Attack Defense Methods Based on Differential Privacy and Data Enhancement. Proceedings of the Communication in Computer and Information Science, Manchester, UK.10.1007/978-981-97-0827-7_23 ↗
54. Ma, Y., Zhu, X., and Hsu, J. (2019). Data Poisoning against Differentially-Private Learners: Attacks and Defences. arXiv, Available online: https://arxiv.org/abs/1903.09860.10.24963/ijcai.2019/657 ↗
55. Machine Learning Security Against Data Poisoning: Are We There Yet?
56. Cheng, Z., Li, Z., Zhang, L., and Zhang, S. (2020, January 2–6). Differentially Private Machine Learning Model against Model Extraction Attack. Proceedings of the IEEE 2020 International Conferences on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics), Rhodes, Greece. Available online: https://ieeexplore.ieee.org/document/9291542.10.1109/iThings-GreenCom-CPSCom-SmartData-Cybermatics50389.2020.00125 ↗
57. Miura, T., Hasegawa, S., and Shibahara, T. (2021). MEGEX: Data-free model extraction attack against gradient-based explainable AI. arXiv, Available online: https://arxiv.org/abs/2107.08909.
58. C2FMI: Corse-to-Fine Black-Box Model Inversion Attack10.1109/TDSC.2023.3285071 ↗
59. Qiu, Y., Yu, H., Fang, H., Yu, W., Chen, B., Wang, X., Xia, S.-T., and Xu, K. (2024). MIBench: A Comprehensive Benchmark for Model Inversion Attack and Defense. arXiv, Available online: https://arxiv.org/abs/2410.05159.
60. Stock, J., Lange, L., Erhard, R., and Federrath, H. (2024, January 18–19). Property Inference as a Regression Problem: Attacks and Defense. Proceedings of the International Conference on Security and Cryptography, Bengaluru, India. Available online: https://www.scitepress.org/publishedPapers/2024/128638/pdf/index.html.
61. Oh, A.H., Belgrave, A., and Cho, K. (2022). A General Framework for Auditing Differentially Private Machine Learning. Advances in Neural Information Processing Systems, The MIT Press. Available online: https://openreview.net/forum?id=AKM3C3tsSx3.
62. Zanella-Béguelin, S., Wutschitz, L., Tople, S., Salem, A., Rühle, V., Paverd, A., Naseri, M., Köpf, B., and Jones, D. (2023, January 23–29). Bayesian Estimation Of Differential Privacy. Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA.
63. Cowan, E., Shoemate, M., and Pereira, M. (2024). Hands-On Differential Privacy, O’Reilly Media, Inc.
64. Differential Privacy: General Inferential Limits via Intervals of Measures
65. Kilpala, M., and Kärkäinen, T. (2024). Artificial Intelligence and Differential Privacy: Review of Protection Estimate Models. Artificial Intelligence for Security: Enhancing Protection in a Changing World, Springer Nature.10.1007/978-3-031-57452-8_3 ↗
66. Balle, B., and Wang, Y.-X. (2018, January 10–15). Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoising. Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, Sweden. Available online: http://proceedings.mlr.press/v80/balle18a/balle18a.pdf.
67. The Bounded Gaussian Mechanism for Differential Privacy
68. Zhang, K., Zhang, Y., Sun, R., Tsai, P.-W., Ul Hassan, M., Yuan, X., Xue, M., and Chen, J. (2024, January 19–23). Bounded and Unbiased Composite Differential Privacy. Proceedings of the IEEE Symposium on Security and Privacy, San Francisco, CA, USA.10.1109/SP54263.2024.00108 ↗
69. Nanayakkara, P., Smart, M.A., Cummings, R., and Kaptchuk, G. (2023, January 9–11). What Are the Chances? Explaining the Epsilon Parameter in Differential Privacy. Proceedings of the 32nd USINEX Security Symposium, Anaheim, CA, USA.
70. Cannone, C., Kamath, G., McMillan, A., Smith, A., and Ullman, J. (2019, January 23–26). The Structure Of Optimal Private Tests For Simple Hypotheses. Proceedings of the Annual ACM Symposium on Theory of Computing, Phoenix, AZ, USA. Available online: https://arxiv.org/abs/1811.11148.10.1145/3313276.3316336 ↗
71. Dwork, C., and Feldman, V. (2018). Privacy-preserving Prediction. arXiv, Available online: https://arxiv.org/abs/1803.10266.
72. Mironov, I. (2017, January 21–25). Rényi Differential Privacy. Proceedings of the 30th IEEE Computer Security Foundations Symposium, CSF, Santa Barbara, CA, USA.10.1109/CSF.2017.11 ↗
73. Evaluating Laplace noise addition to satisfy differential privacy for numeric data
74. Differential privacy scheme using Laplace mechanism and statistical method computation in deep neural network for privacy preservation10.1016/j.engappai.2023.107399 ↗
75. Generalized Gaussian Mechanism for Differential Privacy10.1109/TKDE.2018.2845388 ↗
76. Dong, J., Roth, A., and Su, W.J. (2019). Gaussian Differential privacy. arXiv, Available online: https://arxiv.org/abs/1905.02383.
77. Tight Analysis of Privacy and Utility Tradeoff in Approximate Differential Privacy
78. Whitehouse, J., Ramdas, A., Rogers, R., and Wu, Z.S. (2023). Fully-Adaptive Composition in Differential Privacy. arXiv, Available online: https://arxiv.org/abs/2203.05481.
79. Vaudenay, S. (2006). Our Data, Ourselves: Privacy Via Distributed Noise Generation. Advances in Cryptology—EUROCRYPT, Springer.10.1007/11761679 ↗
80. Zhu, K., Fioretto, F., and Van Hentenryck, P. (2022, January 23–29). Post-processing of Differentially Private Data: A Fairness Perspective. Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI), Vienna, Austria.10.24963/ijcai.2022/559 ↗
81. Ganev, G., Annamalai, M.S.M.S., and De Cristofaro, E. (2024). The Elusive Pursuit of Replicating PATE-GAN: Benchmarking, Auditing, Debugging. arXiv, Available online: https://arxiv.org/abs/2406.13985.
82. Naseri, M., Hayes, J., and De Cristofaro, E. (2022). Local and Central Differential Privacy for Robustness and Privacy in Federated Learning. arXiv, Available online: https://arxiv.org/abs/2009.03561.10.14722/ndss.2022.23054 ↗
83. Babesne, B. (2019). Local Differential Privacy: A tutorial. arXiv, Available online: https://arxiv.org/abs/1907.11908.
84. Nasr, M., Shokri, R., and Houmandsadr, A. (2020). Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning. arXiv, Available online: https://arxiv.org/abs/1812.00910.10.1109/SP.2019.00065 ↗
85. Galli, F., Biswas, S., Jung, K., Cucinotta, T., and Palamidessi, C. (2022). Group privacy for personalized federated learning. arXiv, Available online: https://arxiv.org/abs/2206.03396.
86. Cormode, G., Jha, S., Kulkarni, T., Li, N., Srivastava, D., and Wang, T. (2018, January 10–15). Privacy At Scale: Local Differential Privacy in Practice. Proceedings of the ACM SIGMOD International Conference on Management of Data, Houston, TX, USA.10.1145/3183713.3197390 ↗
87. Local Differential Privacy And Its Applications: A Comprehensive Survey10.1016/j.csi.2023.103827 ↗
88. Local Privacy And Minimax Bounds: Sharp Rates For Probability Estimation
89. Ruan, W., Xu, M., Fang, W., Wang, L., Wang, L., and Han, W. (2023, January 21–25). Private, Efficient, and Accurate: Protecting Models Trained by Multi-party Learning with Differential Privacy. Proceedings of the—IEEE Symposium on Security and Privacy, San Francisco, CA, USA.10.1109/SP46215.2023.10179422 ↗
90. Differential privacy in deep learning: A literature review10.1016/j.neucom.2024.127663 ↗
91. Kang, Y., Liu, Y., Niu, B., Tong, X., Zhang, L., and Wang, W. (2020). Input Perturbation: A New Paradigm between Central and Local Differential Privacy. arXiv, Available online: https://arxiv.org/abs/2002.08570.
92. Differentially Private Empirical Risk Minimization
93. Critical Overview of Privacy in Machine Learning10.1109/MSEC.2021.3076443 ↗
94. Analysis of Application Examples of Differential Privacy in Deep Learning10.1155/2021/4244040 ↗
95. A Survey of Privacy Attacks in Machine Learning
96. Understanding and Defending against White-Box Membership Inference Attack in Deep Learning10.1016/j.knosys.2022.110014 ↗
97. Fang, H., Qiu, Y., Yu, H., Yu, W., Kong, J., Chong, B., Chen, B., Wang, X., and Xia, S.-T. (2024). Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defenses. arXiv, Available online: https://arxiv.org/abs/2402.04013.
98. Study on Choosing the Parameter ε in Differential Privacy
99. Mazzone, F., Al Badawi, A., Polyakov, Y., Everts, M., Hahn, F., and Peter, A. (2024). Investigating Privacy Attacks in the Gray-Box Settings to Enhance Collaborative Learning Schemes. arXiv, Available online: https://arxiv.org/abs/2409.17283.
100. Fredrikson, M., Jha, S., and Ristenpart, T. (2015, January 12–16). Model inversion attacks that exploit confidence information and basic countermeasures. Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, ACM, Denver, CO, USA.10.1145/2810103.2813677 ↗