Federated Distillation Methodology for Label-Based Group Structures
Source abstract
Abstract
In federated learning (FL), clients train models locally without sharing raw data, ensuring data privacy. In particular, federated distillation transfers knowledge to clients regardless of the model architecture. However, when groups of clients with different label distributions exist, sharing the same knowledge among all clients becomes impractical. To address this issue, this paper presents an approach that clusters clients based on the output of a client model trained using their own data. The clients are clustered based on the predictions of their models for each label on a public dataset. Evaluations on MNIST and CIFAR show that our method effectively finds group identities, increasing accuracy by up to 75% over existing methods when the distribution of labels differs significantly between groups. In addition, we observed significant performance improvements on smaller client groups, bringing us closer to fair FL.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · AI in cancer detection · Data Quality and Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Digital Health Care R&D Department, Korea Institute of Industrial Technology, Cheonan 31056, Republic of Korea
Funding
Korea Institute of Industrial Technology · Korea Forest Service
References (26)
Show source-provided references
1. Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A.N., Bonawitz, K., Charles, Z., Cormode, G., and Cummings, R. (2021). Advances and Open Problems in Federated Learning, Foundations and Trends® in Machine Learning.10.1561/2200000083 ↗
2. A survey on federated learning10.1016/j.knosys.2021.106775 ↗
3. The future of digital health with federated learning10.1038/s41746-020-00323-1 ↗
4. Jeong, E., Oh, S., Kim, H., Park, J., Bennis, M., and Kim, S.L. (2018). Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv.
5. Li, D., and Wang, J. (2019). Fedmd: Heterogenous federated learning via model distillation. arXiv.
6. Chang, H., Shejwalkar, V., Shokri, R., and Houmansadr, A. (2019). Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv.
7. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints10.1109/TNNLS.2020.3015958 ↗
8. An efficient framework for clustered federated learning
9. Collaborative federated learning for healthcare: Multi-modal COVID-19 diagnosis at the edge10.1109/OJCS.2022.3206407 ↗
10. Huang, L., Shea, A.L., Qian, H., Masurkar, A., Deng, H., and Liu, D. (2019). Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records. J. Biomed. Inform., 99.10.1016/j.jbi.2019.103291 ↗
11. Mansour, Y., Mohri, M., Ro, J., and Suresh, A.T. (2020). Three approaches for personalization with applications to federated learning. arXiv.
12. Briggs, C., Fan, Z., and Andras, P. (2020, January 19–24). Federated learning with hierarchical clustering of local updates to improve training on non-IID data. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN) IEEE, Glasgow, UK.10.1109/IJCNN48605.2020.9207469 ↗
13. Multi-center federated learning: Clients clustering for better personalization10.1007/s11280-022-01046-x ↗
14. Duan, M., Liu, D., Ji, X., Liu, R., Liang, L., Chen, X., and Tan, Y. (2020). FedGroup: Efficient clustered federated learning via decomposed data-driven measure. arXiv.10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00042 ↗
15. Huang, W., Ye, M., Shi, Z., Li, H., and Du, B. (2023, January 8–22). Rethinking federated learning with domain shift: A prototype view. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) IEEE, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01565 ↗
16. Group knowledge transfer: Federated learning of large cnns at the edge
17. Ensemble distillation for robust model fusion in federated learning
18. Guha, N., Talwalkar, A., and Smith, V. (2019). One-shot federated learning. arXiv.
19. Li, Q., He, B., and Song, D. (2020). Practical one-shot federated learning for cross-silo setting. arXiv.10.24963/ijcai.2021/205 ↗
20. Zhou, Y., Pu, G., Ma, X., Li, X., and Wu, D. (2020). Distilled one-shot federated learning. arXiv.
21. LeCun, Y. (2023, March 21). The MNIST Database of Handwritten Digits. Available online: http://yann.lecun.com/exdb/mnist/.
22. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2017, January 20–22). Communication-efficient learning of deep networks from decentralized data. Proceedings of the Artificial Intelligence and Statistics, PMLR, Fort Lauderdale, FL, USA.
23. Objective criteria for the evaluation of clustering methods10.1080/01621459.1971.10482356 ↗
24. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis10.1016/0377-0427(87)90125-7 ↗
25. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data10.1109/TMC.2021.3070013 ↗
26. Krizhevsky, A., and Hinton, G. (2023, March 21). Learning multiple LAYERS of Features from Tiny Images. Available online: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.