pFedZKD: A One-Shot Personalized Federated Learning Framework via Evolutionary Architecture Search and Data-Free Distillation
Source abstract
Abstract
Personalized federated learning (PFL) faces significant challenges in resource-constrained edge environments, where strict communication budgets and severe system heterogeneity must be jointly addressed. Although one-shot federated learning reduces communication overhead, existing methods typically impose unified model architectures or rely on coarse manual selection strategies, limiting their adaptability to highly heterogeneous data distributions and restricting personalized representation capability. To overcome these limitations, we propose Personalized Federated Zero-shot Knowledge Distillation (pFedZKD), a data-free one-shot federated learning framework designed for structurally heterogeneous scenarios. The framework follows a decouple-and-reconstruct collaborative paradigm. On the client side (decoupling stage), we introduce Particle Swarm Optimization-based Federated Neural Architecture Search (PSO-FedNAS), a gradient-free neural architecture search method that enables each client to autonomously discover a customized convolutional architecture aligned with its local data distribution, eliminating the need for architectural consistency across clients. On the server side (reconstruction stage), to address parameter-space incompatibility caused by structural heterogeneity, we develop an architecture-agnostic multi-teacher zero-shot knowledge distillation mechanism (Multi-ZSKD). This method synthesizes pseudo-samples in latent space to extract semantic consensus from heterogeneous client models and transfers the aggregated knowledge to a unified global student model without accessing real data. The entire collaborative process is completed within a single communication round, substantially reducing communication cost while enhancing privacy preservation. Extensive experiments on MNIST, FashionMNIST, SVHN, and CIFAR-10 under heterogeneous data settings demonstrate that pFedZKD consistently achieves superior personalization accuracy, global generalization performance, and communication efficiency compared with state-of-the-art PFL methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Domain Adaptation and Few-Shot Learning · Advanced Data and IoT Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Control Engineering, China University of Mining, No. 1 Daxue Road, Xuzhou 221116, China
School of Internet of Things Engineering, Wuxi University, No. 333 Xishan Avenue, Wuxi 214105, China
School of Electronic Information Engineering, Huaiyin Institute of Technology, Faculty of Electronic and Information Engineering, No. 1 Meicheng East Road, Huai’an 223003, China
Funding
gang hua · yonggang xu · xuanyang
References (45)
Show source-provided references
1. Advances and Open Problems in Federated Learning10.1561/2200000083 ↗
2. Communication-Efficient Learning of Deep Networks from Decentralized Data
3. Federated Learning for Healthcare Applications10.1109/JIOT.2023.3325822 ↗
4. Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clients10.1109/TMC.2024.3461852 ↗
5. Federated Learning: Challenges, Methods, and Future Directions
6. Towards Personalized Federated Learning10.1109/TNNLS.2022.3160699 ↗
7. Dinh, C.T., Tran, N.H., and Nguyen, T.D. (2022). Personalized Federated Learning with Moreau Envelopes. arXiv.
8. Collins, L., Hassani, H., Mokhtari, A., and Shakkottai, S. (2021). Exploiting Shared Representations for Personalized Federated Learning. arXiv.
9. Fallah, A., Mokhtari, A., and Ozdaglar, A. (2020). Personalized Federated Learning: A Meta-Learning Approach. arXiv.
10. Evolutionary deep learning: A survey10.1016/j.neucom.2022.01.099 ↗
11. A Survey on Evolutionary Neural Architecture Search10.1109/TNNLS.2021.3100554 ↗
12. Liu, H., Simonyan, K., and Yang, Y. (2019). DARTS: Differentiable Architecture Search. arXiv.
13. Akimoto, Y., Shirakawa, S., Yoshinari, N., Uchida, K., Saito, S., and Nishida, K. (2019). Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search. arXiv.
14. Li, L., Khodak, M., Balcan, M.F., and Talwalkar, A. (2021). Geometry-Aware Gradient Algorithms for Neural Architecture Search. arXiv.
15. Lu, Z., Whalen, I., Boddeti, V., Dhebar, Y., Deb, K., Goodman, E., and Banzhaf, W. (2019). NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm. arXiv.10.1145/3321707.3321729 ↗
16. Lu, Z., Deb, K., Goodman, E., Banzhaf, W., and Boddeti, V.N. (2020). NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search. arXiv.10.1007/978-3-030-58452-8_3 ↗
17. Sinha, N., and Chen, K.W. (2020). Evolving Neural Architecture Using One Shot Model. arXiv.10.1145/3449639.3459275 ↗
18. Kennedy, J., and Eberhart, R. (December, January 27). Particle swarm optimization. Proceedings of the ICNN’95—International Conference on Neural Networks, Perth, WA, Australia.
19. A Particle Swarm Optimization-Based Flexible Convolutional Autoencoder for Image Classification10.1109/TNNLS.2018.2881143 ↗
20. Surrogate-Assisted Particle Swarm Optimization for Evolving Variable-Length Transferable Blocks for Image Classification10.1109/TNNLS.2021.3054400 ↗
21. Particle swarm optimization of deep neural networks architectures for image classification10.1016/j.swevo.2019.05.010 ↗
22. He, C., Annavaram, M., and Avestimehr, S. (2021). Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search. arXiv.
23. Dudziak, L., Laskaridis, S., and Fernandez-Marques, J. (2022). FedorAS: Federated Architecture Search under system heterogeneity. arXiv.
24. Zhou, Y., Pu, G., Ma, X., Li, X., and Wu, D. (2021). Distilled One-Shot Federated Learning. arXiv.
25. Zhang, J., Chen, C., Li, B., Lyu, L., Wu, S., Ding, S., Shen, C., and Wu, C. (2022). DENSE: Data-Free One-Shot Federated Learning. arXiv.
26. Heinbaugh, C.E., Luz-Ricca, E., and Shao, H. (2023, January 1–5). Data-Free One-Shot Federated Learning Under Very High Statistical Heterogeneity. Proceedings of the Eleventh International Conference on Learning Representations, Kigali, Rwanda.
27. Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. (2021). FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. arXiv.
28. Li, T., Hu, S., Beirami, A., and Smith, V. (2021). Ditto: Fair and Robust Federated Learning Through Personalization. arXiv.
29. Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020). Federated Optimization in Heterogeneous Networks. arXiv.
30. Liang, P.P., Liu, T., Ziyin, L., Allen, N.B., Auerbach, R.P., Brent, D., Salakhutdinov, R., and Morency, L.P. (2020). Think Locally, Act Globally: Federated Learning with Local and Global Representations. arXiv.
31. Arivazhagan, M.G., Aggarwal, V., Singh, A.K., and Choudhary, S. (2019). Federated Learning with Personalization Layers. arXiv.
32. Smith, V., Chiang, C.K., Sanjabi, M., and Talwalkar, A. (2018). Federated Multi-Task Learning. arXiv.
33. Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints10.1109/TNNLS.2020.3015958 ↗
34. An Efficient Framework for Clustered Federated Learning10.1109/TIT.2022.3192506 ↗
35. Huang, Y., Chu, L., Zhou, Z., Wang, L., Liu, J., Pei, J., and Zhang, Y. (2021). Personalized Cross-Silo Federated Learning on Non-IID Data. arXiv.10.1609/aaai.v35i9.16960 ↗
36. Li, D., and Wang, J. (2019). FedMD: Heterogenous Federated Learning via Model Distillation. arXiv.
37. Lin, T., Kong, L., Stich, S.U., and Jaggi, M. (2021). Ensemble Distillation for Robust Model Fusion in Federated Learning. arXiv.
38. Li, M., Zhang, X., Wang, Q., LIU, T., Wu, R., Wang, W., Zhuang, F., Xiong, H., and Yu, D. (2024, January 10–15). Resource-Aware Federated Self-Supervised Learning with Global Class Representations. Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada.10.52202/079017-0321 ↗
39. Zhu, Z., Hong, J., and Zhou, J. (2021). Data-Free Knowledge Distillation for Heterogeneous Federated Learning. arXiv.
40. Yang, Q., Chen, J., Yin, X., Xie, J., and Wen, Q. (2022, January 22–25). FedMMD: Heterogenous Federated Learning based on Multi-teacher and Multi-feature Distillation. Proceedings of the 2022 7th International Conference on Computer and Communication Systems (ICCCS), Wuhan, China.10.1109/ICCCS55155.2022.9846843 ↗
41. Yao, D., Shi, Y., Liu, T., and Xu, Z. (2025). FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devices. arXiv.10.1145/3774904.3792743 ↗
42. Personalized Federated Learning in One-Shot: A Method for Heterogeneous Data Scenarios10.1109/JIOT.2025.3589414 ↗
43. FedLPA: Personalized One-shot Federated Learning with Layer-Wise Posterior Aggregation
44. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
45. Nayak, G.K., Mopuri, K.R., Shaj, V., Babu, R.V., and Chakraborty, A. (2019). Zero-Shot Knowledge Distillation in Deep Networks. arXiv.