A Secure and Effective Energy-Aware Fixed-Point Quantization Scheme for Asynchronous Federated Learning
Source abstract
Abstract
Asynchronous federated learning (AsynFL) can effectively mitigate the impact of heterogeneity of edge nodes on joint training while satisfying participant user privacy protection and data security. However, the frequent exchange of massive data can lead to excess communication overhead between edge and central nodes regardless of whether the federated learning (FL) algorithm uses synchronous or asynchronous aggregation. Therefore, there is an urgent need for a method that can simultaneously take into account device heterogeneity and edge node energy consumption reduction. This paper proposes a novel Fixed-point Asynchronous Federated Learning (FixedAsynFL) algorithm, which could mitigate the resource consumption caused by frequent data communication while alleviating the effect of device heterogeneity. FixedAsynFL uses fixed-point quantization to compress the local and global models in AsynFL. In order to balance energy consumption and learning accuracy, this paper proposed a quantization scale selection mechanism. This paper examines the mathematical relationship between the quantization scale and energy consumption of the computation/communication process in the FixedAsynFL. Based on considering the upper bound of quantization noise, this paper optimizes the quantization scale by minimizing communication and computation consumption. This paper performs pertinent experiments on the MNIST dataset with several edge nodes of different computing efficiency. The results show that the FixedAsynFL algorithm with an 8-bit quantization can significantly reduce the communication data size by 81.3% and save the computation energy in the training phase by 74.9% without significant loss of accuracy. According to the experimental results, we can see that the proposed AsynFixedFL algorithm can effectively solve the problem of device heterogeneity and energy consumption limitation of edge nodes.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Stochastic Gradient Optimization Techniques · Advanced Data and IoT Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (44)
Show source-provided references
1. 6G-enabled internet of things: Vision, techniques, and open issues10.32604/cmes.2022.021094 ↗
2. Communication-efficient learning of deep networks from decentralized data
3. Federated learning for privacy-preserved medical internet of things10.32604/iasc.2022.023763 ↗
4. Federated learning for internet of things: A comprehensive survey10.1109/COMST.2021.3075439 ↗
5. Federated learning meets blockchain in edge computing: Opportunities and challenges10.1109/JIOT.2021.3072611 ↗
6. Towards asynchronous federated learning for heterogeneous edge-powered internet of things10.1016/j.dcan.2021.04.001 ↗
7. Federated learning via over-the-air computation10.1109/TWC.2019.2961673 ↗
8. Adaptive federated learning in resource constrained edge computing systems10.1109/JSAC.2019.2904348 ↗
9. Toward an intelligent edge: Wireless communication meets machine learning10.1109/MCOM.001.1900103 ↗
10. Hierarchical federated learning across heterogeneous cellular networks
11. Asynchronous federated learning over wireless communication networks10.1109/TWC.2022.3153495 ↗
12. Distributed federated learning for ultra-reliable low-latency vehicular communications10.1109/TCOMM.2019.2956472 ↗
13. C. Xie, S. Koyejo and I. Gupta, “Asynchronous federated optimization,” arXiv preprint arXiv:1903.03934, 2019.
14. C. Xu, Y. Qu, Y. Xiang and L. Gao, “Asynchronous federated learning on heterogeneous devices: A survey,” arXiv preprint arXiv:2109.04269, 2021.
15. In-datacenter performance analysis of a tensor processing unit
16. Predicting parameters in deep learning
17. Low-rank compression of neural nets: Learning the rank of each layer
18. G. Hinton, O. Vinyals and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, vol.2, no. 7, 2015.
19. E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis et al., “Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data,” arXiv preprint arXiv:1811.11479, 2018.
20. S. Han, H. Mao and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149, 2015.
21. Survey of communication overhead of federated learning
22. Deep learning with limited numerical precision
23. Analytical guarantees on numerical precision of deep neural networks
24. Bottou · Advances in Neural Information Processing Systems · 2007 · 20 · 161
25. Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training
26. Noise benefits in backpropagation and deep bidirectional pre-training
27. Asynchronous online federated learning for edge devices with non-iid data
28. Efficient and robust asynchronous federated learning with stragglers
29. Vanhoucke · Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011 · 2011
30. Fixed point quantization of deep convolutional networks
31. Quantization and training of neural networks for efficient integer-arithmetic-only inference
32. Octo: {INT8} training with loss-aware compensation and backward quantization for tiny on-device learning
33. Relaxed quantization for discretized neural networks
34. Learning to quantize deep networks by optimizing quantization intervals with task loss
35. Data-free quantization through weight equalization and bias correction
36. Dynamic aggregation for heterogeneous quantization in federated learning10.1109/TWC.2021.3076613 ↗
37. M. M. Amiri, D. Gunduz, S. R. Kulkarni and H. V. Poor, “Federated learning with quantized global model updates,” arXiv preprint arXiv:2006.10672, 2020.
38. UVeQFed: Universal vector quantization for federated learning10.1109/TSP.2020.3046971 ↗
39. Neural network quantization in federated learning at the edge10.1016/j.ins.2021.06.039 ↗
40. Quantized distributed federated learning for industrial internet of things10.1109/JIOT.2021.3139772 ↗
41. R. Chen, L. Li, K. Xue, C. Zhang, M. Pan et al., “To talk or to work: Energy efficient federated learning over mobile devices via the weight quantization and 5G transmission co-design,” arXiv preprint arXiv:2012.11070, 2012.
42. Petzold · Code: The Hidden Language of Computer Hardware and Software · 2000
43. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition10.1109/TASL.2011.2134090 ↗
44. Gradient-based learning applied to document recognition10.1109/5.726791 ↗