Enhancing Privacy and Communication Efficiency in Federated Learning Through Selective Low-Rank Adaptation and Differential Privacy
Source abstract
Abstract
Federated learning (FL) enables collaborative model training without centralizing raw data, but its application to large-scale vision models remains constrained by high communication cost, data heterogeneity, and privacy risks. Furthermore, in real-world applications such as autonomous driving and healthcare, model updates can inadvertently expose sensitive information even without direct data sharing. This highlights the need for frameworks that balance privacy, efficiency, and accuracy. The current approach to addressing information exposure involves encrypting data by incorporating additional encoding. However, such approaches to encrypting data significantly increase communication costs. In this paper, we propose Federated Share-A Low-Rank Adaptation with Differential Privacy (FedSA-LoRA-DP), a parameter-efficient and privacy-preserving federated learning framework. The framework combines selective aggregation of low-rank parameters with Differential Privacy (DP), ensuring that only lightweight components are shared while formally bounding individual data influence. Since DP simply perturbs the numeric values of existing parameters without altering their dimensionality or structure, it does not increase communication cost. This design allows FedSA-LoRA-DP to provide strong privacy guarantees while maintaining communication efficiency and model accuracy. Experiments on CIFAR-100, MNIST, and SVHN datasets demonstrate that the proposed framework achieves accuracy comparable to non-private counterparts, even under heterogeneous non-independent and identically distributed data and partial client participation. These results demonstrate that integrating differential privacy into low-rank adaptation enables privacy-preserving and communication-efficient federated learning without sacrificing model performance across heterogeneous environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Cryptography and Data Security · Stochastic Gradient Optimization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of System Informatics, Kobe University, Kobe 657-8501, Japan
Funding
JSPS KAKENHI
References (45)
Show source-provided references
1. Nguyen, A., Do, T., Tran, M., Nguyen, B.X., Duong, C., Phan, T., Tjiputra, E., and Tran, Q.D. (2022, January 4–9). Deep Federated Learning for Autonomous Driving. Proceedings of the 2022 IEEE Intelligent Vehicles Symposium (IV), Aachen, Germany.10.1109/IV51971.2022.9827020 ↗
2. Zhang, H., Bosch, J., and Olsson, H.H. (2021, January 18–22). End-to-End Federated Learning for Autonomous Driving Vehicles. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Virtual.10.1109/IJCNN52387.2021.9533808 ↗
3. Konečný, J., McMahan, H.B., Ramage, D., and Richtárik, P. (2016). Federated Optimization: Distributed Machine Learning for On-Device Intelligence. arXiv.
4. Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data10.1109/TNNLS.2019.2944481 ↗
5. Singh, A., and Zhu, J. (2017, January 20–22). Communication-efficient learning of deep networks from decentralized data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA. Proceedings of Machine Learning Research (PMLR).
6. Towards Federated Learning at Scale: System Design
7. Advances and Open Problems in Federated Learning10.1561/2200000083 ↗
8. Chaudhuri, K., and Salakhutdinov, R. (2019, January 9–15). Parameter-Efficient Transfer Learning for NLP. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA. Proceedings of Machine Learning Research (PMLR).
9. Zong, C., Xia, F., Li, W., and Navigli, R. (2021, January 20–23). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online.
10. Moens, M.F., Huang, X., Specia, L., and Yih, S.W.t. (2021, January 7–11). The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.
11. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022). Lora: Low-rank adaptation of large language models. arXiv.
12. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., and Garnett, R. (2019, January 8–14). Deep Leakage from Gradients. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
13. Geiping, J., Bauermeister, H., Dröge, H., and Moeller, M. (2020, January 6–12). Inverting gradients-how easy is it to break privacy in federated learning?. Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS’20), Red Hook, NY, USA.
14. Huang, Y., Gupta, S., Song, Z., Li, K., and Arora, S. (2021, January 6–14). Evaluating gradient inversion attacks and defenses in federated learning. Proceedings of the 35th International Conference on Neural Information Processing Systems (NIPS’21), Red Hook, NY, USA.
15. Zhu, G., Li, D., Gu, H., Han, Y., Yao, Y., Fan, L., and Yang, Q. (2024, January 3–7). FedMIA: An Effective Membership Inference Attack Exploiting “All for One” Principle in Federated Learning. Proceedings of the 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Denver, CO, USA.10.1109/CVPR52734.2025.01922 ↗
16. Wang, X., Wang, N., Wu, L., Guan, Z., Du, X., and Guizani, M. (June, January 28). GBMIA: Gradient-based Membership Inference Attack in Federated Learning. Proceedings of the ICC 2023—IEEE International Conference on Communications, Rome, Italy.10.1109/ICC45041.2023.10279702 ↗
17. Dwork, C., McSherry, F., Nissim, K., and Smith, A. (2006). Calibrating Noise to Sensitivity in Private Data Analysis, Massachusetts Institute of Technology.10.1007/11681878_14 ↗
18. Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., and Zhang, L. (2016, January 24–28). Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS’16), New York, NY, USA.10.1145/2976749.2978318 ↗
19. Mironov, I. (2017, January 22–25). Rényi Differential Privacy. Proceedings of the 2017 IEEE 30th Computer Security Foundations Symposium (CSF), Santa Barbara, CA, USA.10.1109/CSF.2017.11 ↗
20. Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., and Zhao, J. (2021). Opacus: User-friendly differential privacy library in PyTorch. arXiv.
21. Zheng, Q., Chen, S., Long, Q., and Su, W. (2021, January 13–15). Federated f-differential privacy. Proceedings of the International Conference on Artificial Intelligence and Statistics, Virtual.
22. ALDP-FL for Adaptive Local Differential Privacy in Federated Learning10.1038/s41598-025-12575-6 ↗
23. FedFR-ADP: Adaptive Differential Privacy with Feedback Regulation for Robust Model Performance in Federated Learning10.1016/j.inffus.2024.102796 ↗
24. Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. (2020, January 26–30). On the Convergence of FedAvg on Non-IID Data. Proceedings of the 8th International Conference on Learning Representations (ICLR) 2020, Addis Ababa, Ethiopia.
25. Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A.T. (2020, January 13–18). Scaffold: Stochastic controlled averaging for federated learning. Proceedings of the International Conference on Machine Learning, Virtual.
26. Federated Optimization in Heterogeneous Networks
27. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
28. Bernstein, J., Wang, Y.X., Azizzadenesheli, K., and Anandkumar, A. (2018, January 10–15). signSGD: Compressed optimisation for non-convex problems. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
29. QSGD: Communication-efficient SGD via gradient quantization and encoding
30. Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., and Pedarsani, R. (2020, January 26–28). Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. Proceedings of the International Conference on Artificial Intelligence and Statistics, Online.
31. Guo, P., Zeng, S., Wang, Y., Fan, H., Wang, F., and Qu, L. (2025). Selective Aggregation for Low-Rank Adaptation in Federated Learning. arXiv.
32. Sun, Y., Li, Z., Li, Y., and Ding, B. (2024). Improving LoRA in Privacy-preserving Federated Learning. arXiv.
33. Liu, X.Y., Zhu, R., Zha, D., Gao, J., Zhong, S., White, M., and Qiu, M. (2024). Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning. arXiv.10.1145/3682068 ↗
34. Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning10.1145/3682068 ↗
35. A Blockchain-Assisted Trusted Federated Learning for Smart Agriculture10.1007/s42979-025-03672-4 ↗
36. Blockchained On-Device Federated Learning10.1109/LCOMM.2019.2921755 ↗
37. Weng, S., Zhang, L., Feng, D., Feng, C., Wang, R., Klaine, P.V., and Imran, M.A. (2022, January 18–23). Privacy-preserving federated learning based on differential privacy and momentum gradient descent. Proceedings of the 2022 International Joint Conference on Neural Networks (IJCNN), Padua, Italy.10.1109/IJCNN55064.2022.9889795 ↗
38. Ding, J., Zhang, X., Chen, M., Xue, K., Zhang, C., and Pan, M. (2019, January 9–12). Differentially private robust ADMM for distributed machine learning. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA.10.1109/BigData47090.2019.9005716 ↗
39. Jayaraman, B., and Evans, D. (2019, January 14–16). Evaluating differentially private machine learning in practice. Proceedings of the 28th USENIX Security Symposium (USENIX Security 19), Santa Clara, CA, USA.
40. Gaussian differential privacy10.1111/rssb.12454 ↗
41. Zhang, L., Zhang, L., Shi, S., Chu, X., and Li, B. (2023). Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv.
42. Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report.
43. Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. (2020, January 23–28). Big transfer (bit): General visual representation learning. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58558-7_29 ↗
44. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
45. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A.Y. (2011, January 21). Reading Digits in Natural Images with Unsupervised Feature Learning. Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Granada, Spain.