An Overview of Reinforcement Learning Algorithms for Handover Management in 5G Ultra-Dense Small Cell Networks
Source abstract
Abstract
The fifth generation (5G) wireless technology emerged with marvelous effort to state, design, deployment and standardize the upcoming wireless network generation. Artificial intelligence (AI) and machine learning (ML) techniques are well capable to support 5G latest technologies that are expected to deliver high data rate to upcoming use cases and services such as massive machine type communications (mMTC), enhanced mobile broadband (eMBB), and ultra-reliable low latency communications (uRLLC). These services will surely help Gbps of data within the latency of few milliseconds in Internet of Things paradigm. This survey presented 5G mobility management in ultra-dense small cells networks using reinforcement learning techniques. First, we discussed existing surveys then we are focused on handover (HO) management in ultra-dense small cells (UDSC) scenario. Following, this study also discussed how machine learning algorithms can help in different HO scenarios. Nevertheless, future directions and challenges for 5G UDSC networks were concisely addressed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced MIMO Systems Optimization · Advanced Wireless Communication Technologies · Energy Harvesting in Wireless Networks
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Optical Engineering, Sejong University, Seoul 05006, Korea
School of Intelligent Mechatronics Engineering, Sejong University, Seoul 05006, Korea
Funding
National Research Foundation of Korea
References (100+)
Show source-provided references
1. Tikhomirov, A., Omelyanchuk, E., and Semenova, A. (2018, January 14–15). Recommended 5G frequency bands evaluation. Proceedings of the 2018 Systems of Signals Generating and Processing in the Field of on Board Communications, Moscow, Russia.10.1109/SOSG.2018.8350639 ↗
2. Speicher, S., Sirotkin, S., Palat, S., and Davydov, A. (2021). 5G System Overview. 5G Radio Access Network Architecture: The Dark Side of 5G, Wiley.10.1002/9781119550921 ↗
3. 3GPP NR V2X Mode 2: Overview, Models and System-Level Evaluation10.1109/ACCESS.2021.3090855 ↗
4. Coverage and handoff analysis of 5G fractal small cell networks10.1109/TWC.2018.2890662 ↗
5. Sönmez, Ş., Shayea, I., Khan, S.A., and Alhammadi, A. (2020, January 1–2). Handover management for next-generation wireless networks: A brief overview. Proceedings of the 2020 IEEE Microwave Theory and Techniques in Wireless Communications (MTTW), Riga, Latvia.10.1109/MTTW51045.2020.9245065 ↗
6. Millimetre wave frequency band as a candidate spectrum for 5G network architecture: A survey10.1016/j.phycom.2018.11.003 ↗
7. A survey of mmWave user association mechanisms and spectrum sharing approaches: An overview, open issues and challenges, future research trends10.1007/s11276-019-01976-x ↗
8. Coverage, handoff and cost optimization for 5G heterogeneous network10.1016/j.phycom.2020.101037 ↗
9. Terahertz-enabled wireless system for beyond-5G ultra-fast networks: A brief survey10.1109/MNET.2019.1800430 ↗
10. Enabling multi-functional 5G and beyond user equipment: A survey and tutorial10.1109/ACCESS.2019.2936291 ↗
11. Survey on algorithms and strategies for mobility enhancement under heterogeneous network (hetnet) deployment circumstances
12. Applications of deep reinforcement learning in communications and networking: A survey10.1109/COMST.2019.2916583 ↗
13. Althamary, I., Huang, C.W., and Lin, P. (2019, January 24–28). A survey on multi-agent reinforcement learning methods for vehicular networks. Proceedings of the 2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC), Tangier, Morocco.10.1109/IWCMC.2019.8766739 ↗
14. Machine learning paradigms for next-generation wireless networks10.1109/MWC.2016.1500356WC ↗
15. Combination of ultra-dense networks and other 5G enabling technologies: A survey10.1109/ACCESS.2020.2969980 ↗
16. A survey on handover management: From LTE to NR10.1109/ACCESS.2019.2937405 ↗
17. Bithas, P.S., Michailidis, E.T., Nomikos, N., Vouyioukas, D., and Kanatas, A.G. (2019). A survey on machine-learning techniques for UAV-based communications. Sensors, 19.10.3390/s19235170 ↗
18. Next generation 5G wireless networks: A comprehensive survey10.1109/COMST.2016.2532458 ↗
19. A survey on handoffs—Lessons for 60 GHz based wireless systems
20. Lee, Y.L., and Qin, D. (2019, January 18–21). A survey on applications of deep reinforcement learning in resource management for 5G heterogeneous networks. Proceedings of the 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Lanzhou, China.10.1109/APSIPAASC47483.2019.9023331 ↗
21. Mobility management in emerging ultra-dense cellular networks: A survey, outlook, and future research directions10.1109/ACCESS.2020.3027258 ↗
22. Cognition in UAV-aided 5G and beyond communications: A survey10.1109/TCCN.2020.2968311 ↗
23. Deep learning for intelligent wireless networks: A comprehensive survey10.1109/COMST.2018.2846401 ↗
24. Communication and networking technologies for UAVs: A survey10.1016/j.jnca.2020.102739 ↗
25. Big data analytics, machine learning, and artificial intelligence in next-generation wireless networks10.1109/ACCESS.2018.2837692 ↗
26. Survey on artificial intelligence techniques in 5G networks
27. Multi-agent reinforcement learning based resource management in MEC-and UAV-assisted vehicular networks10.1109/JSAC.2020.3036962 ↗
28. Hazareena, A., and Mustafa, B.A. (2018, January 29–31). A survey: On the waveforms for 5G. Proceedings of the 2018 Second International Conference on Electronics, Communication and Aerospace Technology (ICECA), Coimbatore, India.10.1109/ICECA.2018.8474641 ↗
29. Pandi, V.S., and Priya, J.L. (2017, January 21–22). A survey on 5G mobile technology. Proceedings of the 2017 IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI), Chennai, India.10.1109/ICPCSI.2017.8391995 ↗
30. Millimeter and THz wave for 5G and beyond
31. Khurpade, J.M., Rao, D., and Sanghavi, P.D. (2018, January 5). A Survey on IOT and 5G Network. Proceedings of the 2018 International Conference on Smart City and Emerging Technology (ICSCET), Mumbai, India.10.1109/ICSCET.2018.8537340 ↗
32. A survey on 5G massive MIMO localization10.1016/j.dsp.2019.05.005 ↗
33. A survey on key enabling technologies towards 5G10.1088/1757-899X/1033/1/012011 ↗
34. A survey of 5G technologies: Regulatory, standardization and industrial perspectives10.1016/j.dcan.2017.09.010 ↗
35. Cayamcela, M.E.M., and Lim, W. (2018, January 17–19). Artificial intelligence in 5G technology: A survey. Proceedings of the 2018 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Korea.
36. 5G D2D networks: Techniques, challenges, and future prospects10.1109/JSYST.2017.2773633 ↗
37. A survey of advanced techniques for spectrum sharing in 5G networks10.1109/MWC.2017.1700069 ↗
38. Key management for beyond 5G mobile small cells: A survey10.1109/ACCESS.2019.2914359 ↗
39. Survey of radio resource management in 5G heterogeneous networks10.1109/ACCESS.2020.3002252 ↗
40. Survey on terahertz nanocommunication and networking: A top-down perspective10.1109/JSAC.2021.3071837 ↗
41. Mihovska, A. (2020, January 20–22). Small cell deployment challenges in ultradense networks: Architecture and resource management. Proceedings of the 2020 12th International Symposium on Communication Systems, Networks and Digital Signal Processing (CSNDSP), Porto, Portugal.
42. A comprehensive survey of RAN architectures toward 5G mobile communication system10.1109/ACCESS.2019.2919657 ↗
43. Kasim, A.N. (2020). A survey mobility management in 5G networks. arXiv.
44. Next generation wireless cellular networks: Ultra-dense multi-tier and multi-cell cooperation perspective10.1007/s11276-018-1796-y ↗
45. A comprehensive survey: Small cell meets massive MIMO10.1016/j.phycom.2017.11.004 ↗
46. López-Pérez, D., De Domenico, A., Piovesan, N., Baohongqiang, H., Xinli, G., Qitao, S., and Debbah, M. (2021). A survey on 5G energy efficiency: Massive MIMO, lean carrier design, sleep modes, and machine learning. arXiv.10.1109/COMST.2022.3142532 ↗
47. Energy-Efficient MAC for Cellular IoT: State-of-the-Art, Challenges, and Standardization10.1109/TGCN.2021.3062093 ↗
48. Power saving techniques for 5G and beyond10.1109/ACCESS.2020.3001180 ↗
49. RRC State Handling for 5G10.1109/MCOM.2018.1700957 ↗
50. Survey of radio resource management issues and proposals for energy-efficient cellular networks that will cover billions of machines10.1186/s13638-016-0636-y ↗
51. A survey of 5G network systems: Challenges and machine learning approaches10.1007/s13042-020-01178-4 ↗
52. Samal, S.R., Bandopadhaya, S., Swain, K., and Poulkov, V. (2021). Mobility Management in Heterogeneous Cellular Networks. J. Mob. Multimed., 407–426.10.13052/jmm1550-4646.171321 ↗
53. Multi-criteria handover mobility management in 5G cellular network10.1016/j.comcom.2021.04.020 ↗
54. A survey of 5G network: Architecture and emerging technologies10.1109/ACCESS.2015.2461602 ↗
55. On-demand ultra-dense cloud drone networks: Opportunities, challenges and benefits10.1109/MCOM.2018.1701001 ↗
56. Becvar, Z., Vondra, M., Mach, P., Plachy, J., and Gesbert, D. (2017, January 17–19). Performance of mobile networks with UAVs: Can flying base stations substitute ultra-dense small cells?. Proceedings of the European Wireless 2017: 23th European Wireless Conference, Dresden, Germany.
57. Fakhreddine, A., Bettstetter, C., Hayat, S., Muzaffar, R., and Emini, D. (2019, January 21). Handover challenges for cellular-connected drones. Proceedings of the 5th Workshop on Micro Aerial Vehicle Networks, Systems, and Applications, Seoul, Korea.10.1145/3325421.3329770 ↗
58. Intelligent Massive Traffic Handling Scheme in 5G Bottleneck Backhaul Networks
59. Tezergil, B., and Onur, E. (2021). Wireless Backhaul in 5G and Beyond: Issues, Challenges and Opportunities. arXiv.10.1109/COMST.2022.3203578 ↗
60. Mobility management solutions for 5G networks: Architecture and services10.1016/j.comnet.2019.107082 ↗
61. Autonomic handover management for heterogeneous networks in a future internet context: A survey10.1109/COMST.2019.2916188 ↗
62. Handover Management of Drones in Future Mobile Networks: 6G Technologies10.1109/ACCESS.2021.3051097 ↗
63. Understanding UAV cellular communications: From existing networks to massive MIMO10.1109/ACCESS.2018.2876700 ↗
64. Ivancic, W.D., Kerczewski, R.J., Murawski, R.W., Matheou, K., and Downey, A.N. (2019, January 9–11). Flying drones beyond visual line of sight using 4g LTE: Issues and concerns. Proceedings of the 2019 Integrated Communications, Navigation and Surveillance Conference (ICNS), Washington, DC, USA.10.1109/ICNSURV.2019.8735246 ↗
65. Survey on UAV cellular communications: Practical aspects, standardization advancements, regulation, and security challenges10.1109/COMST.2019.2906228 ↗
66. A survey on space-aerial-terrestrial integrated 5G networks10.1016/j.comnet.2020.107212 ↗
67. A survey on cellular-connected UAVs: Design challenges, enabling 5G/B5G innovations, and experimental advancements10.1016/j.comnet.2020.107451 ↗
68. Small cells in the forthcoming 5G/IoT: Traffic modelling and deployment overview
69. Platzgummer, V., Raida, V., Krainz, G., Svoboda, P., Lerch, M., and Rupp, M. (2019, January 22–25). UAV-based coverage measurement method for 5G. Proceedings of the 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall), Honolulu, HI, USA.10.1109/VTCFall.2019.8891252 ↗
70. Huang, W., Zhang, H., and Zhou, M. (2019, January 8–11). Analysis of handover probability based on equivalent model for 3D UAV networks. Proceedings of the 2019 IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), Istanbul, Turkey.10.1109/PIMRC.2019.8904320 ↗
71. Sanusi, J., Aibinu, A.M., Adeshina, S., Koyunlu, G., and Idris, S. (2019, January 23–24). Review of Handover in Li-Fi and Wi-Fi Networks. Proceedings of the International Conference on Computer Networks and Inventive Communication Technologies, Coimbatore, India.10.1007/978-3-030-37051-0_107 ↗
72. Xu, Y., Gui, G., Gacanin, H., and Adachi, F. (2021). A survey on resource allocation for 5G heterogeneous networks: Current research, future trends and challenges. IEEE Commun. Surv. Tutor., 668–695.10.1109/COMST.2021.3059896 ↗
73. User-centric ultra-dense networks for 5G: Challenges, methodologies, and directions10.1109/MWC.2016.7462488 ↗
74. Interoperability in heterogeneous wireless networks using fis-enn vertical handover model10.1007/s11277-019-06406-x ↗
75. Selvi, M.P., and Sendhilnathan, S. (2016). Fuzzy based Mobility management in 4G wireless Networks. Braz. Arch. Biol. Technol., 59.10.1590/1678-4324-2016161047 ↗
76. Chang, S., Li, G., Zhao, J., Wu, J., and Zhang, B. (2021, January 15–17). Individualized optimization of handover parameters in 5G (NR) indoor and outdoor co-frequency scenarios. Proceedings of the 2021 IEEE International Conference on Consumer Electronics and Computer Engineering (ICCECE), Guangzhou, China.10.1109/ICCECE51280.2021.9342278 ↗
77. Chattate, I., Bakkoury, J., Khiat, A., and El Khaili, M. (2016, January 24–26). Overview on technology of vertical handover and MIH architecture. Proceedings of the 2016 4th IEEE International Colloquium on Information Science and Technology (CiSt), Tangier, Morocco.10.1109/CIST.2016.7805069 ↗
78. Enabling vertical handover decisions in heterogeneous wireless networks: A state-of-the-art and a classification10.1109/SURV.2013.082713.00141 ↗
79. Yi, F., Fu, W., and Liang, H. (2018, January 2–6). Model-based reinforcement learning: A survey. Proceedings of the International Conference on Electronic Business (ICEB), Guilin, China.
80. Machine learning algorithms for wireless sensor networks: A survey10.1016/j.inffus.2018.09.013 ↗
81. Reinforcement learning algorithms: Survey and classification10.17485/ijst/2017/v10i1/109385 ↗
82. Reinforcement Learning-Based Optimization for Drone Mobility in 5G and Beyond Ultra-Dense Networks
83. Van Otterlo, M., and Wiering, M. (2012). Reinforcement learning and markov decision processes. Reinforcement Learning, Springer.10.1007/978-3-642-27645-3_1 ↗
84. A survey on semi-supervised learning10.1007/s10994-019-05855-6 ↗
85. A survey on deep learning: Algorithms, techniques, and applications
86. Kemi Victoria Dada, Nachaat AbdElatif Mohamed, and Humaira Arshad
87. Arulkumaran, K., Deisenroth, M.P., Brundage, M., and Bharath, A.A. (2017). A brief survey of deep reinforcement learning. arXiv.10.1109/MSP.2017.2743240 ↗
88. Ankile, L.L., Heggland, M.F., and Krange, K. (2020). Deep Convolutional Neural Networks: A survey of the foundations, selected improvements, and some current applications. arXiv.
89. Tang, C., Chen, X., Chen, Y., and Li, Z. (2018, January 11–13). A MDP-based network selection scheme in 5G ultra-dense network. Proceedings of the 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS), Singapore.10.1109/PADSW.2018.8645006 ↗
90. Mohammed, T., Albeshri, A., Katib, I., and Mehmood, R. (2020). UbiPriSEQ—Deep reinforcement learning to manage privacy, security, energy, and QoS in 5G IoT hetnets. Appl. Sci., 10.10.3390/app10207120 ↗
91. Deep reinforcement learning for dynamic uplink/downlink resource allocation in high mobility 5G HetNet10.1109/JSAC.2020.3005495 ↗
92. AI-enabled future wireless networks: Challenges, opportunities, and open issues10.1109/MVT.2019.2919236 ↗
93. Future intelligent and secure vehicular network toward 6G: Machine-learning approaches10.1109/JPROC.2019.2954595 ↗
94. Deep reinforcement learning: A brief survey10.1109/MSP.2017.2743240 ↗
95. Liu, J., Gu, X., and Liu, S. (2019). Reinforcement learning with world model. arXiv.
96. A survey of preference-based reinforcement learning methods
97. Nguyen, M.T., and Kwon, S. (2021). Machine Learning–Based Mobility Robustness Optimization Under Dynamic Cellular Networks. IEEE Access, 77830–77844.10.1109/ACCESS.2021.3083554 ↗
98. Cognitive cellular networks: A Q-learning framework for self-organizing networks10.1109/TNSM.2016.2522080 ↗
99. Mobility robustness optimization and load balancing in self-organized cellular networks: Towards cognitive network management10.3233/JIFS-191558 ↗
100. Koda, Y., Yamamoto, K., Nishio, T., and Morikura, M. (2018, January 15–19). Reinforcement learning based predictive handover for pedestrian-aware mmWave networks. Proceedings of the IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), Honolulu, HI, USA.10.1109/INFCOMW.2018.8406993 ↗