Federated Reinforcement Learning in IoT: Applications, Opportunities and Open Challenges
Source abstract
Abstract
The internet of things (IoT) represents a disruptive concept that has been changing society in several ways. There have been several successful applications of IoT in the industry. For example, in transportation systems, the novel internet of vehicles (IoV) concept has enabled new research directions and automation solutions. Moreover, reinforcement learning (RL), federated learning (FL), and federated reinforcement learning (FRL) have demonstrated remarkable success in solving complex problems in different applications. In recent years, new solutions have been developed based on this combined framework (i.e., federated reinforcement learning). Conversely, there is a lack of analysis concerning IoT applications and a standard view of challenges and future directions of the current FRL landscape. Thereupon, the main goal of this research is to present a literature review of federated reinforcement learning (FRL) applications in IoT from multiple perspectives. We focus on analyzing applications in multiple areas (e.g., security, sustainability and efficiency, vehicular solutions, and industrial services) to highlight existing solutions, their characteristics, and research gaps. Additionally, we identify key short- and long-term challenges leading to new opportunities in the field. This research intends to picture the current FRL ecosystem in IoT to foster the development of new solutions based on existing challenges.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Mobile Crowdsensing and Crowdsourcing · Transportation and Mobility Innovations
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Canadian Institute for Cybersecurity (CIC), University of New Brunswick (UNB), Fredericton, NB E3B 5A3, Canada
References (100+)
Show source-provided references
1. The internet of things: An overview
2. Tan, L., and Wang, N. (2010, January 20–22). Future internet: The internet of things. Proceedings of the 3rd International Conference on Advanced Computer Theory and Engineering (ICACTE), Chengdu, China.
3. An overview of internet of vehicles10.1109/CC.2014.6969789 ↗
4. Smart logistics based on the internet of things technology: An overview10.1080/13675567.2020.1757053 ↗
5. Ramlowat, D.D., and Pattanayak, B.K. Exploring the internet of things (IoT) in education: A review. Proceedings of the Information Systems Design and Intelligent Applications: Proceedings of 5th International Conference INDIA 2018.10.1007/978-981-13-3338-5_23 ↗
6. Verdouw, C., Wolfert, S., and Tekinerdogan, B. (2016). Internet of things in agriculture. CABI Rev., 1–12.10.1079/PAVSNNR201611035 ↗
7. Pan, J., and Yang, Z. (2018, January 19–21). Cybersecurity challenges and opportunities in the new “edge computing + IoT” world. Proceedings of the ACM International Workshop on Security in Software Defined Networks & Network Function Virtualization, Tempe, AZ, USA.10.1145/3180465.3180470 ↗
8. Internet of things (IoT) security dataset evolution: Challenges and future directions10.1016/j.iot.2023.100780 ↗
9. Danso, P.K., Neto, E.C.P., Dadkhah, S., Zohourian, A., Molyneaux, H., and Ghorbani, A.A. (2022, January 10–12). Ensemble-based Intrusion Detection for internet of things Devices. Proceedings of the IEEE 19th International Conference on Smart Communities: Improving Quality of Life Using ICT, IoT and AI (HONET), Marietta, GE, USA.10.1109/HONET56683.2022.10019140 ↗
10. Industrial internet of things business models in the machine-to-machine context10.1016/j.indmarman.2019.08.008 ↗
11. A lightweight cellular automata based encryption technique for IoT applications10.1109/ACCESS.2019.2906326 ↗
12. Perceived Internet privacy concerns on social networks in Europe10.1016/j.techfore.2015.01.021 ↗
13. Singh, R., Dwivedi, A.D., Srivastava, G., Chatterjee, P., and Lin, J.C.W. (2023). A Privacy Preserving internet of things Smart Healthcare Financial System. IEEE Internet Things J., Early Access.10.1109/JIOT.2022.3233783 ↗
14. A game theoretic approach for privacy preserving model in IoT-based transportation10.1109/TITS.2018.2885054 ↗
15. Sutton, R.S. (1998, January 24–27). Reinforcement learning: Past, present and future. Proceedings of the Simulated Evolution and Learning: Second Asia-Pacific Conference on Simulated Evolution and Learning, SEAL’98, Canberra, Australia.
16. Sutton, R.S. (1999, January 29–31). Open theoretical questions in reinforcement learning. Proceedings of the Computational Learning Theory: 4th European Conference, EuroCOLT’99, Nordkirchen, Germany.
17. Federated learning: Challenges, methods, and future directions
18. Jin, H., Peng, Y., Yang, W., Wang, S., and Zhang, Z. (2022, January 28–30). Federated reinforcement learning with environment heterogeneity. Proceedings of the International Conference on Artificial Intelligence and Statistics, Virtual Conference.
19. A Selective federated reinforcement learning Strategy for Autonomous Driving
20. A resource-constrained and privacy-preserving edge-computing-enabled clinical decision system: A federated reinforcement learning approach10.1109/JIOT.2021.3057653 ↗
21. Wang, H., Kaplan, Z., Niu, D., and Li, B. (2020, January 6–9). Optimizing federated learning on non-iid data with reinforcement learning. Proceedings of the IEEE INFOCOM 2020—IEEE Conference on Computer Communications, Online.10.1109/INFOCOM41043.2020.9155494 ↗
22. Multiagent federated reinforcement learning for Secure Incentive Mechanism in Intelligent Cyber–Physical Systems10.1109/JIOT.2021.3081626 ↗
23. Qi, J., Zhou, Q., Lei, L., and Zheng, K. (2021). Federated reinforcement learning: Techniques, applications, and open challenges. arXiv.10.20517/ir.2021.02 ↗
24. Reinforcement learning
25. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning10.1016/S0004-3702(99)00052-1 ↗
26. Policy gradient methods for reinforcement learning with function approximation
27. Reinforcement learning: The good, the bad and the ugly10.1016/j.conb.2008.08.003 ↗
28. Li, Y. (2017). Deep reinforcement learning: An overview. arXiv.
29. Scaling reinforcement learning toward RoboCup soccer
30. Reinforcement learning: A survey10.1613/jair.301 ↗
31. Kurach, K., Raichuk, A., Stańczyk, P., Zając, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., and Bousquet, O. (2020, January 7–12). Google research football: A novel reinforcement learning environment. Proceedings of the AAAI Conference on artificial intelligence, Hilton, NY, USA.10.1609/aaai.v34i04.5878 ↗
32. Zhang, H., Feng, S., Liu, C., Ding, Y., Zhu, Y., Zhou, Z., Zhang, W., Yu, Y., Jin, H., and Li, Z. (2019, January 13–17). Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario. Proceedings of the World Wide Web Conference, San Francisco, CA, USA.10.1145/3308558.3314139 ↗
33. Tizhoosh, H.R. (2005, January 14–16). Reinforcement learning based on actions and opposite actions. Proceedings of the International Conference on Artificial Intelligence and Machine Learning, Hong Kong, China.
34. Branavan, S.R., Chen, H., Zettlemoyer, L., and Barzilay, R. (2009, January 2–7). Reinforcement learning for mapping instructions to actions. Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, Singapore.10.3115/1687878.1687892 ↗
35. Matignon, L., Laurent, G.J., and Le Fort-Piat, N. (2006, January 10–14). Reward function and initial values: Better choices for accelerated goal-directed reinforcement learning. Proceedings of the Artificial Neural Networks–ICANN 2006: 16th International Conference, Athens, Greece.10.1007/11840817_87 ↗
36. Convergence results for single-step on-policy reinforcement-learning algorithms10.1023/A:1007678930559 ↗
37. Fast reinforcement learning with generalized policy updates10.1073/pnas.1907370117 ↗
38. Data science in the Research Domain Criteria era: Relevance of machine learning to the study of stress pathology, recovery, and resilience10.1177/2470547017747553 ↗
39. The path planning of mobile robot by neural networks and hierarchical reinforcement learning10.3389/fnbot.2020.00063 ↗
40. Q-learning10.1007/BF00992698 ↗
41. Bianchi, R.A., Ros, R., and Lopez de Mantaras, R. (2009, January 20–23). Improving reinforcement learning by using case based heuristics. Proceedings of the Case-Based Reasoning Research and Development: 8th International Conference on Case-Based Reasoning, ICCBR 2009, Seattle, WA, USA.10.1007/978-3-642-02998-1_7 ↗
42. An energy-aware and Q-learning-based area coverage for oil pipeline monitoring systems using sensors and internet of things10.1038/s41598-022-12181-w ↗
43. Q-learning aided resource allocation and environment recognition in LoRaWAN with CSMA/CA10.1109/ACCESS.2019.2948111 ↗
44. Fan, J., Wang, Z., Xie, Y., and Yang, Z. (2020, January 10–11). A theoretical analysis of deep Q-learning. Proceedings of the Learning for Dynamics and Control PMLR, Palo Alto, CA, USA.
45. Brim, A. (2020, January 6–8). Deep reinforcement learning pairs trading with a double deep Q-network. Proceedings of the 10th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.10.1109/CCWC47524.2020.9031159 ↗
46. A new deep-Q-learning-based transmission scheduling mechanism for the cognitive internet of things10.1109/JIOT.2017.2759728 ↗
47. Smart packet transmission scheduling in cognitive IoT systems: DDQN based approach10.1109/ACCESS.2022.3168549 ↗
48. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv.
49. Li, S., Bing, S., and Yang, S. (2018). Distributional advantage actor–critic. arXiv.
50. Peng, B., Li, X., Gao, J., Liu, J., Chen, Y.N., and Wong, K.F. (2018, January 15–20). Adversarial advantage actor–critic model for task-completion dialogue policy learning. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461918 ↗
51. Chen, G., Xu, X., Zeng, Q., and Zhang, Y.D. (2022). A Vehicle-Assisted Computation Offloading Algorithm Based on Proximal Policy Optimization in Vehicle Edge Networks. Mob. Netw. Appl., 1–15.10.1007/s11036-022-02029-y ↗
52. Deep-Graph-Based reinforcement learning for Joint Cruise Control and Task Offloading for Aerial Edge internet of things (EdgeIoT)10.1109/JIOT.2022.3182119 ↗
53. Deep deterministic policy gradient (DDPG)-based energy harvesting wireless communications10.1109/JIOT.2019.2921159 ↗
54. Intrusion detection in green internet of things: A deep deterministic policy gradient-based algorithm10.1109/TGCN.2021.3073714 ↗
55. Babaeizadeh, M., Frosio, I., Tyree, S., Clemons, J., and Kautz, J. (2016). Reinforcement learning through asynchronous advantage actor–critic on a gpu. arXiv.
56. Towards distributed and autonomous IoT service placement in fog computing using asynchronous advantage actor–critic algorithm
57. Advances and open problems in federated learning10.1561/2200000083 ↗
58. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2017, January 20–22). Communication-efficient learning of deep networks from decentralized data. Proceedings of the artificial intelligence and Statistics, PMLR, Fort Lauderdale, FL, USA.
59. FedSky: An efficient and privacy-preserving scheme for federated mobile crowdsensing10.1109/JIOT.2021.3109058 ↗
60. Efficient and privacy-enhanced federated learning for industrial artificial intelligence10.1109/TII.2019.2945367 ↗
61. Federated learning for data privacy preservation in vehicular cyber-physical systems10.1109/MNET.011.1900317 ↗
62. Blockchained on-device federated learning10.1109/LCOMM.2019.2921755 ↗
63. PEFL: Deep Privacy-Encoding based federated learning Framework for Smart Agriculture10.1109/MM.2021.3112476 ↗
64. Federated Machine Learning: Concept and Applications10.1145/3339474 ↗
65. Architectural patterns for the design of federated learning systems10.1016/j.jss.2022.111357 ↗
66. Liu, Y., Kang, Y., Zou, T., Pu, Y., He, Y., Ye, X., Ouyang, Y., Zhang, Y.Q., and Yang, Q. (2022). Vertical federated learning. arXiv.
67. Chen, T., Jin, X., Sun, Y., and Yin, W. (2020). Vafl: A method of vertical asynchronous federated learning. arXiv.
68. Liu, Y., Zhang, X., and Wang, L. (2020). Asymmetrical vertical federated learning. arXiv.
69. Gao, D., Ju, C., Wei, X., Liu, Y., Chen, T., and Yang, Q. (2019). Hhhfl: Hierarchical heterogeneous horizontal federated learning for electroencephalography. arXiv.
70. Breaking the centralized barrier for cross-device federated learning
71. TrustFed: A framework for fair and trustworthy cross-device federated learning in IIoT10.1109/TII.2021.3075706 ↗
72. A practical cross-device federated learning framework over 5g networks10.1109/MWC.005.2100435 ↗
73. Huang, C., Huang, J., and Liu, X. (2022). Cross-Silo federated learning: Challenges and Opportunities. arXiv.
74. Huang, Y., Chu, L., Zhou, Z., Wang, L., Liu, J., Pei, J., and Zhang, Y. (2021, January 2–9). Personalized cross-silo federated learning on non-iid data. Proceedings of the AAAI Conference on artificial intelligence, Online.10.1609/aaai.v35i9.16960 ↗
75. Jiang, Z., Wang, W., and Liu, Y. (2021). Flashe: Additively symmetric homomorphic encryption for cross-silo federated learning. arXiv.
76. Zhang, Y., Zeng, D., Luo, J., Xu, Z., and King, I. (2023). A Survey of Trustworthy federated learning with Perspectives on Security, Robustness, and Privacy. arXiv.10.1145/3543873.3587681 ↗
77. Trustworthy federated learning via blockchain10.1109/JIOT.2022.3201117 ↗
78. Nguyen, T.D., Marchal, S., Miettinen, M., Fereidooni, H., Asokan, N., and Sadeghi, A.R. (2019, January 7–9). DÏoT: A federated self-learning anomaly detection system for IoT. Proceedings of the IEEE 39th International conference on distributed computing systems (ICDCS), Dallas, TX, USA.10.1109/ICDCS.2019.00080 ↗
79. Federated-learning-based anomaly detection for iot security attacks10.1109/JIOT.2021.3077803 ↗
80. Zhang, C., Li, M., and Wu, D. (2022). Federated Multidomain Learning With Graph Ensemble Autoencoder GMM for Emotion Recognition. IEEE Trans. Intell. Transp. Syst., Early Access.10.1109/TITS.2022.3203800 ↗
81. Hamid, O.H. (2023). Data-Centric and Model-Centric AI: Twin Drivers of Compact and Robust Industry 4.0 Solutions. Appl. Sci., 13.10.3390/app13052753 ↗
82. Hamid, O.H., and Braun, J. (2017, January 1–3). Reinforcement learning and attractor neural network models of associative learning. Proceedings of the Computational Intelligence: 9th International Joint Conference, IJCCI 2017, Funchal, Portugal.
83. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., and Dunning, I. (2018, January 25–31). Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. Proceedings of the International Conference on Machine Learning, PMLR, Vienna, Austria.
84. Hoffman, M.W., Shahriari, B., Aslanides, J., Barth-Maron, G., Momchev, N., Sinopalnikov, D., Stańczyk, P., Ramos, S., Raichuk, A., and Vincent, D. (2020). Acme: A research framework for distributed reinforcement learning. arXiv.
85. Tolpegin, V., Truex, S., Gursoy, M.E., and Liu, L. (2020, January 14–18). Data poisoning attacks against federated learning systems. Proceedings of the Computer Security–ESORICS 2020: 25th European Symposium on Research in Computer Security, ESORICS 2020, Guildford, UK. Part I 25.10.1007/978-3-030-58951-6_24 ↗
86. Lim, H.K., Kim, J.B., Heo, J.S., and Han, Y.H. (2020). Federated reinforcement learning for training control policies on multiple IoT devices. Sensors, 20.10.3390/s20051359 ↗
87. Liang, X., Liu, Y., Chen, T., Liu, M., and Yang, Q. (2022). Federated and Transfer Learning, Springer.
88. A federated reinforcement learning framework for incumbent technologies in beyond 5G networks10.1109/MNET.011.2000611 ↗
89. Rjoub, G., Bentahar, J., and Wahab, O.A. (2022, January 19–23). Explainable AI-based federated deep reinforcement learning for Trusted Autonomous Driving. Proceedings of the 2022 International Wireless Communications and Mobile Computing (IWCMC), Marrakesh, Morocco.10.1109/IWCMC55113.2022.9824617 ↗
90. Na, S., Krajník, T., Lennox, B., and Arvin, F. (2022). Federated reinforcement learning for Collective Navigation of Robotic Swarms. arXiv.10.1109/TCDS.2023.3239815 ↗
91. Otoum, S., Guizani, N., and Mouftah, H. (2021, January 4–23). Federated reinforcement learning-supported IDS for IoT-steered healthcare systems. Proceedings of the ICC 2021-IEEE International Conference on Communications, Virtual.10.1109/ICC42927.2021.9500698 ↗
92. Federated deep reinforcement learning-Based Spectrum Access Algorithm With Warranty Contract in Intelligent Transportation Systems10.1109/TITS.2022.3179442 ↗
93. Tiwari, P., Lakhan, A., Jhaveri, R.H., and Gronli, T.M. (2023). Consumer-Centric Internet of Medical Things for Cyborg Applications based on federated reinforcement learning. IEEE Trans. Consum. Electron., Early Access.10.1109/TCE.2023.3242375 ↗
94. Dynamic spectrum access for internet-of-things based on federated deep reinforcement learning10.1109/TVT.2022.3166535 ↗
95. Blockchain and federated learning for 5G beyond10.1109/MNET.011.1900598 ↗
96. Devote: Criticality-aware federated service provisioning in fog-based iot environments10.1109/JIOT.2021.3049326 ↗
97. C-fdrl: Context-aware privacy-preserving offloading through federated deep reinforcement learning in cloud-enabled IoT10.1109/TII.2022.3149335 ↗
98. Federated deep reinforcement learning based secure data sharing for internet of things10.1016/j.comnet.2021.108327 ↗
99. Exploring Deep-Reinforcement-Learning-Assisted federated learning for Online Resource Allocation in Privacy-Preserving EdgeIoT10.1109/JIOT.2022.3176739 ↗
100. Anwar, A., and Raychowdhury, A. (2021). Multi-task federated reinforcement learning with adversaries. arXiv.