Survey of Deep Learning-Based Methods for FMCW Radar Odometry and Ego-Localization
Source abstract
Abstract
This paper provides an in-depth review of deep learning techniques to address the challenges of odometry and global ego-localization using frequency modulated continuous wave (FMCW) radar sensors. In particular, we focus on the prediction of odometry, which involves the determination of the ego-motion of a system by external sensors, and loop closure detection, which concentrates on the determination of the ego-position typically on an existing map. We initially emphasize the significance of these tasks in the context of radar sensors and underscore the motivations behind them. The subsequent sections delve into the practical implementation of deep learning approaches, strategically designed to effectively address the aforementioned challenges. We primarily focus on spinning and automotive radar configurations within the domain of autonomous driving. Additionally, we introduce publicly available datasets that have been instrumental in addressing these challenges and analyze the importance and struggles of current methods used for radar based odometry and localization. In conclusion, this paper highlights the distinctions between the addressed tasks and other radar perception applications, while also discussing their differences from challenges posed by alternative sensor modalities. The findings contribute to the ongoing discourse on advancing radar sensor capabilities through the application of deep learning methodologies, particularly in the context of enhancing odometry and ego-localization for autonomous driving applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Target Tracking and Data Fusion in Sensor Networks · Indoor and Outdoor Localization Technologies · Advanced Optical Sensing Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute for Technologies and Management of Digtial Transformation, University of Wuppertal, Gaußstraße 20, 42119 Wuppertal, Germany
References (100+)
Show source-provided references
1. Bavle, H., Sanchez-Lopez, J.L., Cimarelli, C., Tourani, A., and Voos, H. (2023). From slam to situational awareness: Challenges and survey. Sensors, 23.10.3390/s23104849 ↗
2. Performance of laser and radar ranging devices in adverse environmental conditions10.1002/rob.20310 ↗
3. An integrated GNSS/LiDAR-SLAM pose estimation framework for large-scale map building in partially GNSS-denied environments10.1109/TIM.2020.2987049 ↗
4. Yao, S., Guan, R., Huang, X., Li, Z., Sha, X., Yue, Y., Lim, E.G., Seo, H., Man, K.L., and Zhu, X. (2023). Radar-camera fusion for object detection and semantic segmentation in autonomous driving: A comprehensive review. arXiv.10.1109/TIV.2023.3307157 ↗
5. Harlow, K., Jang, H., Barfoot, T.D., Kim, A., and Heckman, C. (2023). A New Wave in Robotics: Survey on Recent mmWave Radar Applications in Robotics. arXiv.
6. Zhang, J., Xie, Y., Ling, L., and Folkesson, J. (2023). A Fully-automatic Side-scan Sonar SLAM Framework. arXiv.
7. Survey on deep learning based computer vision for sonar imagery10.1016/j.engappai.2022.105157 ↗
8. Chen, C., Wang, B., Lu, C.X., Trigoni, N., and Markham, A. (2020). A survey on deep learning for localization and mapping: Towards the age of spatial machine intelligence. arXiv.
9. Zhou, Y., Liu, L., Zhao, H., López-Benítez, M., Yu, L., and Yue, Y. (2022). Towards deep radar perception for autonomous driving: Datasets, methods, and challenges. Sensors, 22.10.3390/s22114208 ↗
10. Analysis of multipath and DOA detection using a fully polarimetric automotive radar10.1017/S1759078718000284 ↗
11. Simultaneous localization and mapping: Part I10.1109/MRA.2006.1678144 ↗
12. Simultaneous localization and mapping (SLAM): Part II10.1109/MRA.2006.1678144 ↗
13. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv.10.18653/v1/2020.acl-main.703 ↗
14. Deep learning for computer vision: A brief review10.1155/2018/7068349 ↗
15. Sarlin, P.E., DeTone, D., Malisiewicz, T., and Rabinovich, A. (2020, January 20–25). Superglue: Learning feature matching with graph neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR42600.2020.00499 ↗
16. Zhu, A.Z., Liu, W., Wang, Z., Kumar, V., and Daniilidis, K. (2018). Robustness meets deep learning: An end-to-end hybrid pipeline for unsupervised learning of egomotion. arXiv.
17. Liu, X., Chen, S.W., Aditya, S., Sivakumar, N., Dcunha, S., Qu, C., Taylor, C.J., Das, J., and Kumar, V. (2018, January 1–5). Robust fruit counting: Combining deep learning, tracking, and structure from motion. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8594239 ↗
18. Sarlin, P.E., Cadena, C., Siegwart, R., and Dymczyk, M. (2019, January 15–20). From coarse to fine: Robust hierarchical localization at large scale. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01300 ↗
19. Filatov, A., Rykov, A., and Murashkin, V. (August, January 31). Any motion detector: Learning class-agnostic scene dynamics from a sequence of lidar point clouds. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9196716 ↗
20. Mohammadi, M.E., Watson, D.P., and Wood, R.L. (2019). Deep learning-based damage detection from aerial SfM point clouds. Drones, 3.10.3390/drones3030068 ↗
21. Im, J.U., Ki, S.W., and Won, J.H. (2024). Omni Point: 3D LiDAR-based Feature Extraction Method for Place Recognition and Point Registration. IEEE Trans. Intell. Veh., 1–18.10.1109/TIV.2023.3348525 ↗
22. O’Shea, K., and Nash, R. (2015). An introduction to convolutional neural networks. arXiv.
23. A review of recurrent neural networks: LSTM cells and network architectures10.1162/neco_a_01199 ↗
24. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.10.3115/v1/D14-1179 ↗
25. Graph neural networks: A review of methods and applications10.1016/j.aiopen.2021.01.001 ↗
26. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
27. Zhang, Y., Wu, Y., Tong, K., Chen, H., and Yuan, Y. (2023). Review of Visual Simultaneous Localization and Mapping Based on Deep Learning. Remote Sens., 15.10.3390/rs15112740 ↗
28. Deep Learning Techniques for Visual SLAM: A Survey10.1109/ACCESS.2023.3249661 ↗
29. Neural Network-Based Recent Research Developments in SLAM for Autonomous Ground Vehicles: A Review10.1109/JSEN.2023.3273913 ↗
30. Comparative Analysis of Radar and Lidar Technologies for Automotive Applications10.1109/MITS.2022.3162886 ↗
31. A tutorial on graph-based SLAM10.1109/MITS.2010.939925 ↗
32. 4-D SLAM: An efficient dynamic bayes network-based approach for dynamic scene understanding10.1109/ACCESS.2020.3042339 ↗
33. Ru, X., Gu, N., Shang, H., and Zhang, H. (2022). MEMS inertial sensor calibration technology: Current status and future trends. Micromachines, 13.10.3390/mi13060879 ↗
34. Rublee, E., Rabaud, V., Konolige, K., and Bradski, G. (2011, January 6–13). ORB: An efficient alternative to SIFT or SURF. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126544 ↗
35. Distinctive image features from scale-invariant keypoints10.1023/B:VISI.0000029664.99615.94 ↗
36. Adolfsson, D., Magnusson, M., Alhashimi, A., Lilienthal, A.J., and Andreasson, H. (2020–24, January 24). Cfear radarodometry-conservative filtering for efficient and accurate radar odometry. Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS51168.2021.9636253 ↗
37. Segal, A., Haehnel, D., and Thrun, S. (July, January 28). Generalized-icp. Proceedings of the Robotics: Science and Systems, Seattle, WA, USA.
38. Fast and robust iterative closest point
39. Overview of the RANSAC Algorithm
40. The normal distributions transform: A new approach to laser scan matching10.1109/IROS.2003.1249285 ↗
41. Kellner, D., Barjenbruch, M., Klappstein, J., Dickmann, J., and Dietmayer, K. (2013, January 6–9). Instantaneous ego-motion estimation using doppler radar. Proceedings of the 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), The Hague, The Netherlands.10.1109/ITSC.2013.6728341 ↗
42. Kellner, D., Barjenbruch, M., Klappstein, J., Dickmann, J., and Dietmayer, K. (June, January 31). Instantaneous ego-motion estimation using multiple Doppler radars. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China.10.1109/ICRA.2014.6907064 ↗
43. Bow3d: Bag of words for real-time loop closing in 3d lidar slam10.1109/LRA.2022.3221336 ↗
44. Louback da Silva Lubanco, D., Schlechter, T., Pichler-Scheder, M., and Kastl, C. (2022, January 20–25). Survey on Radar Odometry. Proceedings of the Computer Aided Systems Theory–EUROCAST 2022: 18th International Conference, Las Palmas de Gran Canaria, Spain. Revised Selected Papers.10.1007/978-3-031-25312-6_73 ↗
45. Deep-learning for radar: A survey10.1109/ACCESS.2021.3119561 ↗
46. A survey of the state-of-the-art localization techniques and their potentials for autonomous vehicle applications10.1109/JIOT.2018.2812300 ↗
47. Arshad, S., and Kim, G.W. (2021). Role of deep learning in loop closure detection for visual and lidar slam: A survey. Sensors, 21.10.3390/s21041243 ↗
48. A survey of machine learning techniques for indoor localization and navigation systems10.1007/s10846-021-01327-z ↗
49. Yin, P., Zhao, S., Cisneros, I., Abuduweili, A., Huang, G., Milford, M., Liu, C., Choset, H., and Scherer, S. (2022). General Place Recognition Survey: Towards the Real-world Autonomy Age. arXiv.
50. Xu, X., Zhang, L., Yang, J., Cao, C., Wang, W., Ran, Y., Tan, Z., and Luo, M. (2022). A review of multi-sensor fusion slam systems based on 3D LIDAR. Remote Sens., 14.10.3390/rs14122835 ↗
51. Camera, LiDAR and multi-modal SLAM systems for autonomous ground vehicles: A survey10.1007/s10846-022-01582-8 ↗
52. Huang, L. (2021, January 18–20). Review on LiDAR-based SLAM techniques. Proceedings of the 2021 International Conference on Signal Processing and Machine Learning (CONF-SPML), Beijing, China.10.1109/CONF-SPML54095.2021.00040 ↗
53. Khan, M.U., Zaidi, S.A.A., Ishtiaq, A., Bukhari, S.U.R., Samer, S., and Farman, A. (2021, January 15–17). A comparative survey of lidar-slam and lidar based sensor technologies. Proceedings of the 2021 Mohammad Ali Jinnah University International Conference on Computing (MAJICC), Karachi, Pakistan.10.1109/MAJICC53071.2021.9526266 ↗
54. Chen, W., Shang, G., Ji, A., Zhou, C., Wang, X., Xu, C., Li, Z., and Hu, K. (2022). An overview on visual slam: From tradition to semantic. Remote Sens., 14.10.3390/rs14133010 ↗
55. Visual place recognition: A survey10.1109/TRO.2015.2496823 ↗
56. Approaches, challenges, and applications for deep visual odometry: Toward complicated and emerging areas10.1109/TCDS.2020.3038898 ↗
57. Deep learning for visual SLAM in transportation robotics: A review10.1093/tse/tdz019 ↗
58. A survey on active simultaneous localization and mapping: State of the art and new frontiers10.1109/TRO.2023.3248510 ↗
59. Favorskaya, M.N. (2023). Deep Learning for Visual SLAM: The State-of-the-Art and Future Trends. Electronics, 12.10.3390/electronics12092006 ↗
60. A survey on visual navigation for artificial agents with deep reinforcement learning10.1109/ACCESS.2020.3011438 ↗
61. A Survey on Transformers for Point Cloud Processing: An Updated Overview10.1109/ACCESS.2022.3198999 ↗
62. Yi, Z. (2024, February 28). A Survey of Radar Perception—Datasets, Methods and Applications. Available online: https://github.com/ZHOUYI1023/awesome-radar-perception.
63. Barnes, D., Gadd, M., Murcutt, P., Newman, P., and Posner, I. (August, January 31). The oxford radar robotcar dataset: A radar extension to the oxford robotcar dataset. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9196884 ↗
64. Boreas: A multi-season autonomous driving dataset10.1177/02783649231160195 ↗
65. Kim, G., Park, Y.S., Cho, Y., Jeong, J., and Kim, A. (August, January 31). Mulran: Multimodal range dataset for urban place recognition. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9197298 ↗
66. Sheeny, M., De Pellegrin, E., Mukherjee, S., Ahrabian, A., Wang, S., and Wallace, A. (June, January 30). RADIATE: A radar dataset for automotive perception in bad weather. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9562089 ↗
67. Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13–19). nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01164 ↗
68. Multi-class road user detection with 3+ 1D radar in the View-of-Delft dataset10.1109/LRA.2022.3147324 ↗
69. ColoRadar: The direct 3D millimeter wave radar dataset10.1177/02783649211068535 ↗
70. Are we ready for unmanned surface vehicles in inland waterways? The usvinland multisensor dataset and benchmark10.1109/LRA.2021.3067271 ↗
71. Mostajabi, M., Wang, C.M., Ranjan, D., and Hsyu, G. (2020, January 13–19). High-resolution radar dataset for semi-supervised learning of dynamic objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00058 ↗
72. Yan, Z., Sun, L., Krajník, T., and Ruichek, Y. (2020, January 25–29). EU long-term dataset with multiple sensors for autonomous driving. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341406 ↗
73. Meyer, M., and Kuschk, G. (2019, January 2–4). Automotive radar dataset for deep learning based 3d object detection. Proceedings of the 2019 16th European Radar Conference (EuRAD), Paris, France.
74. K-Radar: 4D radar object detection for autonomous driving in various weather conditions
75. Rebut, J., Ouaknine, A., Malik, W., and Pérez, P. (2022, January 18–24). Raw high-definition radar for multi-task learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01651 ↗
76. Automotive radar–investigation of mutual interference mechanisms10.5194/ars-8-55-2010 ↗
77. Bi-Level l 1 Optimization-Based Interference Reduction for Millimeter Wave Radars10.1109/TITS.2022.3215636 ↗
78. 4D Radar-based Pose Graph SLAM with Ego-velocity Pre-integration Factor10.1109/LRA.2023.3292574 ↗
79. Zhang, J., Zhuge, H., Wu, Z., Peng, G., Wen, M., Liu, Y., and Wang, D. (June, January 29). 4DRadarSLAM: A 4D Imaging Radar SLAM System for Large-scale Environments based on Pose Graph Optimization. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), Londin, UK.10.1109/ICRA48891.2023.10160670 ↗
80. Khan, K., Rehman, S.U., Aziz, K., Fong, S., and Sarasvady, S. (2014, January 17–19). DBSCAN: Past, present and future. Proceedings of the Fifth International Conference on the Applications of Digital Information and Web Technologies (ICADIWT 2014), Chennai, India.10.1109/ICADIWT.2014.6814687 ↗
81. Relative radar cross section based feature identification with millimeter wave radar for outdoor slam10.1109/IROS.2004.1389389 ↗
82. Predicting millimeter wave radar spectra for autonomous navigation10.1109/JSEN.2009.2037013 ↗
83. Wang, S., Clark, R., Wen, H., and Trigoni, N. (June, January 29). Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989236 ↗
84. Aldera, R., De Martini, D., Gadd, M., and Newman, P. (2019, January 20–24). Fast radar motion estimation with a learnt focus of attention using weak supervision. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8794014 ↗
85. Barnes, D., Weston, R., and Posner, I. (2019). Masking by moving: Learning distraction-free radar odometry from pose information. arXiv.
86. Weston, R., Gadd, M., De Martini, D., Newman, P., and Posner, I. (2022, January 23–27). Fast-MbyM: Leveraging Translational Invariance of the Fourier Transform for Efficient and Accurate Radar Odometry. Proceedings of the 2022 International Conference on Robotics and Automation (ICRA), Philadelphia, PA, USA.10.1109/ICRA46639.2022.9812063 ↗
87. Barnes, D., and Posner, I. (August, January 31). Under the radar: Learning to predict robust keypoints for odometry estimation and metric localisation in radar. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9196835 ↗
88. Burnett, K., Yoon, D.J., Schoellig, A.P., and Barfoot, T.D. (2021). Radar odometry combining probabilistic estimation and unsupervised feature learning. arXiv.10.15607/RSS.2021.XVII.029 ↗
89. Self-supervised scene flow estimation with 4-d automotive radar10.1109/LRA.2022.3187248 ↗
90. Efficient Deep-Learning 4D Automotive Radar Odometry Method10.1109/TIV.2023.3311102 ↗
91. Milli-RIO: Ego-motion estimation with low-cost millimetre-wave radar10.1109/JSEN.2020.3023243 ↗
92. Cai, K., Wang, B., and Lu, C.X. (2021). Autoplace: Robust place recognition with low-cost single-chip automotive radar. arXiv.10.1109/ICRA46639.2022.9811869 ↗
93. Săftescu, Ş., Gadd, M., De Martini, D., Barnes, D., and Newman, P. (August, January 31). Kidnapped radar: Topological radar localisation using rotationally-invariant metric learning. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9196682 ↗
94. Gadd, M., De Martini, D., and Newman, P. (2020, January 20–23). Look around you: Sequence-based radar place recognition with learned rotational invariance. Proceedings of the 2020 IEEE/ION Position, Location and Navigation Symposium (PLANS), Portland, OR, USA.10.1109/PLANS46316.2020.9109951 ↗
95. Gadd, M., De Martini, D., and Newman, P. (2021, January 6–10). Contrastive learning for unsupervised radar place recognition. Proceedings of the 2021 20th International Conference on Advanced Robotics (ICAR), Ljubljana, Slovenia.10.1109/ICAR53236.2021.9659335 ↗
96. Usuelli, M., Frosi, M., Cudrano, P., Mentasti, S., and Matteucci, M. (2023). RadarLCD: Learnable Radar-based Loop Closure Detection Pipeline. arXiv.
97. Yuan, J., Newman, P., and Gadd, M. (2023). Off the Radar: Uncertainty-Aware Radar Place Recognition with Introspective Querying and Map Maintenance. arXiv.10.1109/IROS55552.2023.10341965 ↗
98. Wang, W., de Gusmao, P.P., Yang, B., Markham, A., and Trigoni, N. (June, January 30). Radarloc: Learning to relocalize in fmcw radar. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9560858 ↗
99. Rsl-net: Localising in satellite images from a radar on the ground10.1109/LRA.2020.2965907 ↗
100. Deep learning-based robust positioning for all-weather autonomous driving10.1038/s42256-022-00520-5 ↗