Vision-Based Autonomous Vehicle Systems Based on Deep Learning: A Systematic Literature Review
Source abstract
Abstract
In the past decade, autonomous vehicle systems (AVS) have advanced at an exponential rate, particularly due to improvements in artificial intelligence, which have had a significant impact on social as well as road safety and the future of transportation systems. However, the AVS is still far away from mass production because of the high cost of sensor fusion and a lack of combination of top-tier solutions to tackle uncertainty on roads. To reduce sensor dependency and to increase manufacturing along with enhancing research, deep learning-based approaches could be the best alternative for developing practical AVS. With this vision, in this systematic review paper, we broadly discussed the literature of deep learning for AVS from the past decade for real-life implementation in core fields. The systematic review on AVS implementing deep learning is categorized into several modules that cover activities including perception analysis (vehicle detection, traffic signs and light identification, pedestrian detection, lane and curve detection, road object localization, traffic scene analysis), decision making, end-to-end controlling and prediction, path and motion planning and augmented reality-based HUD, analyzing research works from 2011 to 2021 that focus on RGB camera vision. The literature is also analyzed for final representative outcomes as visualization in augmented reality-based head-up display (AR-HUD) with categories such as early warning, road markings for improved navigation and enhanced safety with overlapping on vehicles and pedestrians in extreme visual conditions to reduce collisions. The contribution of the literature review includes detailed analysis of current state-of-the-art deep learning methods that only rely on RGB camera vision rather than complex sensor fusion. It is expected to offer a pathway for the rapid development of cost-efficient and more secure practical autonomous vehicle systems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Autonomous Vehicle Technology and Safety · Advanced Neural Network Applications · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Center for Artificial Intelligence Technology, Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, Bangi 43600, Malaysia
Funding
Universiti Kebangsaan Malaysia
References (100+)
Show source-provided references
1. A systematic literature review of the factors influencing the adoption of autonomous driving10.1007/s13198-020-00961-4 ↗
2. Three-dimensional path planning for unmanned aerial vehicles using glowworm swarm optimization algorithm10.1007/s13198-017-0663-z ↗
3. Examination of trust and sustainability concerns in autonomous vehicle adoption10.1016/j.techsoc.2020.101361 ↗
4. Driving fatigue among long distance heavy vehicle drivers in Klang Valley, Malaysia10.4028/www.scientific.net/AMM.663.567 ↗
5. Towards guidelines and verification methods for automated vehicle HMIs10.1016/j.trf.2018.10.012 ↗
6. Impacts of gradual automated vehicle penetration on motorway operation: A comprehensive evaluation10.1186/s12544-019-0375-3 ↗
7. Large-scale mapping in complex field scenarios using an autonomous car10.1016/j.eswa.2015.10.045 ↗
8. Autonomous vehicles: Scientometric and bibliometric review10.1080/01441647.2018.1518937 ↗
9. Maurer, M., Christian, G., Lenz, B., and Winner, H. (2016). Autonomous Driving: Technical, Legal and Social Aspects, Springer Nature.10.1007/978-3-662-48847-8 ↗
10. Levinson, J., Askeland, J., Becker, J., Dolson, J., Held, D., Kammel, S., Kolter, J.Z., Langer, D., Pink, O., and Pratt, V. (2011, January 5–9). Towards fully autonomous driving: Systems and algorithms. Proceedings of the 2011 IEEE Intelligent Vehicles Symposium (IV), Baden-Baden, Germany.10.1109/IVS.2011.5940562 ↗
11. Yu, H., Yang, S., Gu, W., and Zhang, S. (2017, January 11–14). Baidu driving dataset and end-to-end reactive control model. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.10.1109/IVS.2017.7995742 ↗
12. Towards autonomous vehicle implementation: Issues and opportunities
13. Design of a multi-sensor cooperation travel environment perception system for autonomous vehicle10.3390/s120912386 ↗
14. A sensor-fusion drivable-region and lane-detection system for autonomous vehicle navigation in challenging road scenarios10.1109/TVT.2013.2281199 ↗
15. Pedestrian Detection using Triple Laser Range Finders
16. Pedestrian recognition and tracking using 3D LiDAR for autonomous vehicle10.1016/j.robot.2016.11.014 ↗
17. Map-based localization method for autonomous vehicles using 3D-LIDAR
18. Computation and Sensor Offloading for Cloud-Based Infrastructure-Assisted Autonomous Vehicles10.1109/JSYST.2019.2959703 ↗
19. Detection and tracking of pedestrians and vehicles using roadside LiDAR sensors10.1016/j.trc.2019.01.007 ↗
20. Huval, B., Wang, T., Tandon, S., Kiske, J., Song, W., Pazhayampallil, J., Andriluka, M., Rajpurkar, P., Migimatsu, T., and Cheng, Y.R. (2015). An empirical evaluation of deep learning on highway driving. arXiv.
21. Leon, F., and Gavrilescu, M. (2019). A review of tracking, prediction and decision making methods for autonomous driving. arXiv.
22. Artificial intelligence applications in the development of autonomous vehicles: A survey10.1109/JAS.2020.1003021 ↗
23. A survey of motion planning and control techniques for self-driving urban vehicles10.1109/TIV.2016.2578706 ↗
24. A survey of autonomous driving: Common practices and emerging technologies10.1109/ACCESS.2020.2983149 ↗
25. Planning and decision-making for autonomous vehicles10.1146/annurev-control-060117-105157 ↗
26. Autonomous vehicle perception: The technology of today and tomorrow10.1016/j.trc.2018.02.012 ↗
27. Self-driving cars: A survey10.1016/j.eswa.2020.113816 ↗
28. Convolutional neural networks for 5G-enabled intelligent transportation system: A systematic review10.1016/j.comcom.2020.01.058 ↗
29. Zhang, Y., Qi, Y., Liu, J., and Wang, Y. (2018). Decade of Vision-Based Pedestrian Detection for Self-Driving: An Experimental Survey and Evaluation, SAE Publishing. 0148-7191.10.4271/2018-01-1603 ↗
30. Ni, J., Chen, Y., Chen, Y., Zhu, J., Ali, D., and Cao, W. (2020). A survey on theories and applications for self-driving cars based on deep learning methods. Appl. Sci., 10.10.3390/app10082749 ↗
31. Fayyad, J., Jaradat, M.A., Gruyer, D., and Najjaran, H. (2020). Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review. Sensors, 20.10.3390/s20154220 ↗
32. Deep reinforcement learning for autonomous driving: A survey10.1109/TITS.2021.3054625 ↗
33. A survey of deep learning techniques for autonomous driving10.1002/rob.21918 ↗
34. Hu, H.-N., Cai, Q.-Z., Wang, D., Lin, J., Sun, M., Krahenbuhl, P., Darrell, T., and Yu, F. (November, January 27). Joint monocular 3D vehicle detection and tracking. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
35. Sang, J., Wu, Z., Guo, P., Hu, H., Xiang, H., Zhang, Q., and Cai, B. (2018). An improved YOLOv2 for vehicle detection. Sensors, 18.10.3390/s18124272 ↗
36. Shakya, V., and Makwana, R.R.S. (2017, January 11–12). Feature selection based intrusion detection system using the combination of DBSCAN, K-Mean++ and SMO algorithms. Proceedings of the 2017 International Conference on Trends in Electronics and Informatics (ICEI), Tirunelveli, India.10.1109/ICOEI.2017.8300843 ↗
37. Learning to detect vehicles by clustering appearance patterns10.1109/TITS.2015.2409889 ↗
38. A novel model based on AdaBoost and deep CNN for vehicle classification10.1109/ACCESS.2018.2875525 ↗
39. Bautista, C.M., Dy, C.A., Mañalac, M.I., Orbe, R.A., and Cordel, M. (2016, January 9–11). Convolutional neural network for vehicle detection in low resolution traffic videos. Proceedings of the 2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia.10.1109/TENCONSpring.2016.7519418 ↗
40. Hierarchical scheme of vehicle detection and tracking in nighttime urban environment10.1007/s12239-018-0035-y ↗
41. SINet: A scale-insensitive convolutional neural network for fast vehicle detection10.1109/TITS.2018.2838132 ↗
42. Wang, Y., Liu, Z., and Deng, W. (2019). Anchor generation optimization and region of interest assignment for vehicle detection. Sensors, 19.10.3390/s19051089 ↗
43. Vehicle type detection based on deep learning in traffic scene10.1016/j.procs.2018.04.281 ↗
44. Vehicle Detection and Ranging Using Two Different Focal Length Cameras
45. Leung, H.K., Chen, X.-Z., Yu, C.-W., Liang, H.-Y., Wu, J.-Y., and Chen, Y.-L. (2019). A Deep-Learning-Based Vehicle Detection Approach for Insufficient and Nighttime Illumination Conditions. Appl. Sci., 9.10.3390/app9224769 ↗
46. Fast branch convolutional neural network for traffic sign recognition10.1109/MITS.2017.2709780 ↗
47. Jung, S., Lee, U., Jung, J., and Shim, D.H. (2016, January 19–22). Real-time Traffic Sign Recognition system with deep convolutional neural network. Proceedings of the 2016 13th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), Xi’an, China.10.1109/URAI.2016.7734014 ↗
48. Cao, J., Song, C., Peng, S., Xiao, F., and Song, S. (2019). Improved traffic sign detection and recognition algorithm for intelligent vehicles. Sensors, 19.10.3390/s19184021 ↗
49. Traffic sign recognition using weighted multi-convolutional neural network10.1049/iet-its.2018.5171 ↗
50. Hole-based traffic sign detection method for traffic signs with red rim10.1007/s00371-013-0879-0 ↗
51. Zhang, J., Huang, M., Jin, X., and Li, X. (2017). A real-time chinese traffic sign detection algorithm based on modified YOLOv2. Algorithms, 10.10.3390/a10040127 ↗
52. Traffic-light sign recognition using Capsule network10.1007/s11042-020-10455-x ↗
53. Bach, M., Stumper, D., and Dietmayer, K. (2018, January 4–7). Deep convolutional traffic light recognition for automated driving. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.10.1109/ITSC.2018.8569522 ↗
54. Weber, M., Wolf, P., and Zöllner, J.M. (2016, January 19–22). DeepTLR: A single deep convolutional network for detection and classification of traffic lights. Proceedings of the 2016 IEEE Intelligent Vehicles Symposium (IV), Gothenburg, Sweden.10.1109/IVS.2016.7535408 ↗
55. Liu, X., Neuyen, M., and Yan, W.Q. (2020). Vehicle-related scene understanding using deep learning. Asian Conference on Pattern Recognition, Springer.10.1007/978-981-15-3651-9_7 ↗
56. Accurate traffic light detection using deep neural network with focal regression loss10.1016/j.imavis.2019.04.003 ↗
57. Behrendt, K., Novak, L., and Botros, R. (June, January 29). A deep learning approach to traffic lights: Detection, tracking, and classification. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989163 ↗
58. Yanagisawa, M., Swanson, E., and Najm, W.G. (2014). Target Crashes and Safety Benefits Estimation Methodology for Pedestrian Crash Avoidance/Mitigation Systems.
59. Autonomous Emergency Brake (AEB) for Pedestrian for ASEAN NCAP Safety Rating Consideration: A Review
60. Angelova, A., Krizhevsky, A., and Vanhoucke, V. (2015, January 26–30). Pedestrian detection with a large-field-of-view deep network. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.10.1109/ICRA.2015.7139256 ↗
61. Zhan, H., Liu, Y., Cui, Z., and Cheng, H. (2019, January 27–30). Pedestrian Detection and Behavior Recognition Based on Vision. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.10.1109/ITSC.2019.8917264 ↗
62. Qiu, Z., Zhao, N., Zhou, L., Wang, M., Yang, L., Fang, H., He, Y., and Liu, Y. (2020). Vision-based moving obstacle detection and tracking in paddy field using improved yolov3 and deep SORT. Sensors, 20.10.3390/s20154082 ↗
63. Ghosh, S., Amon, P., Hutter, A., and Kaup, A. (2017, January 17–20). Reliable pedestrian detection using a deep neural network trained on pedestrian counts. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.10.1109/ICIP.2017.8296368 ↗
64. Wang, X., Jung, C., and Hero, A.O. (2017, January 5–9). Part-level fully convolutional networks for pedestrian detection. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952560 ↗
65. Multispectral pedestrian detection based on deep convolutional neural networks10.1016/j.infrared.2018.08.029 ↗
66. Yamada, K. (2016, January 4–8). Pedestrian detection with a resolution-aware convolutional network. Proceedings of the 2016 23rd International Conference on Pattern Recognition (ICPR), Cancun, Mexico.10.1109/ICPR.2016.7899698 ↗
67. Zhang, J., Xiao, J., Zhou, C., and Peng, C. (2018, January 31). A multi-class pedestrian detection network for distorted pedestrians. Proceedings of the 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), Wuhan, China.10.1109/ICIEA.2018.8397871 ↗
68. Dong, P., and Wang, W. (2016, January 27–30). Better region proposals for pedestrian detection with R-CNN. Proceedings of the 2016 Visual Communications and Image Processing (VCIP), Chengdu, China.10.1109/VCIP.2016.7805452 ↗
69. Nighttime driving: Visual, lighting and visibility challenges10.1111/opo.12659 ↗
70. Overview of wrong-way driving fatal crashes in the United States
71. Traffic accidents in Iraq: An analytical study
72. Ding, B., Liu, Z., and Sun, Y. (2018, January 19–21). Pedestrian Detection in Haze Environments Using Dark Channel Prior and Histogram of Oriented Gradient. Proceedings of the 2018 Eighth International Conference on Instrumentation & Measurement, Computer, Communication and Control (IMCCC), Harbin, China.10.1109/IMCCC.2018.00211 ↗
73. An advanced single-image visibility restoration algorithm for real-world hazy scenes10.1109/TIE.2014.2364798 ↗
74. Deep learning approaches on pedestrian detection in hazy weather10.1109/TIE.2019.2945295 ↗
75. Ju, M., Luo, H., Wang, Z., Hui, B., and Chang, Z. (2019). The application of improved YOLO V3 in multi-scale target detection. Appl. Sci., 9.10.3390/app9183775 ↗
76. Ground Plane Context Aggregation Network for Day-and-Night on Vehicular Pedestrian Detection10.1109/TITS.2020.2991848 ↗
77. Accurate pedestrian detection by human pose regression10.1109/TIP.2019.2942686 ↗
78. Pedestrian detection for autonomous vehicle using multi-spectral cameras10.1109/TIV.2019.2904389 ↗
79. Alvarez, J.M., Gevers, T., LeCun, Y., and Lopez, A.M. (2012). Road scene segmentation from a single image. European Conference on Computer Vision, Springer.10.1007/978-3-642-33786-4_28 ↗
80. Combination of modified U-Net and domain adaptation for road detection10.1049/iet-ipr.2018.6696 ↗
81. Deep Neural Network for Structural Prediction and Lane Detection in Traffic Scene10.1109/TNNLS.2016.2522428 ↗
82. Bayesian curved lane estimation for autonomous driving10.1007/s12652-020-01688-7 ↗
83. Yang, S., Wu, J., Shan, Y., Yu, Y., and Zhang, S. (2019). A Novel Vision-Based Framework for Real-Time Lane Detection and Tracking, SAE Technical Paper.10.4271/2019-01-0690 ↗
84. Mongkonyong, P., Nuthong, C., Siddhichai, S., and Yamakita, M. (2017, January 12–15). Lane detection using randomized Hough transform. Proceedings of the 8th TSME-International Conference on Mechanical Engineering, Bangkok, Thailand.
85. A robust vision-based lane boundaries detection approach for intelligent vehicles10.3923/itj.2012.1184.1192 ↗
86. A novel fpga implementation of hough transform for straight lane detection
87. Dong, Y., Patil, S., van Arem, B., and Farah, H. (2021). A Hybrid Spatial-temporal Deep Learning Architecture for Lane Detection. arXiv.10.1111/mice.12829 ↗
88. Liu, L., Chen, X., Zhu, S., and Tan, P. (2021). CondLaneNet: A Top-to-down Lane Detection Framework Based on Conditional Convolution. arXiv.10.1109/ICCV48922.2021.00375 ↗
89. Dorj, B., Hossain, S., and Lee, D.-J. (2020). Highly Curved Lane Detection Algorithms Based on Kalman Filter. Appl. Sci., 10.10.3390/app10072372 ↗
90. Robust multi-lane detection and tracking using adaptive threshold and lane classification10.1007/s00138-018-0977-0 ↗
91. CNN based lane detection with instance segmentation in edge-cloud computing10.1186/s13677-020-00172-z ↗
92. Lane detection method based on lane structural analysis and CNNs10.1049/iet-its.2017.0143 ↗
93. Robust lane detection from continuous driving scenes using deep neural networks10.1109/TVT.2019.2949603 ↗
94. Real-time road surface and semantic lane estimation using deep features10.1007/s11760-018-1264-2 ↗
95. Chen, P.-R., Lo, S.-Y., Hang, H.-M., Chan, S.-W., and Lin, J.-J. (2018, January 19–21). Efficient road lane marking detection with deep learning. Proceedings of the 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP), Shanghai, China.10.1109/ICDSP.2018.8631673 ↗
96. Ghafoorian, M., Nugteren, C., Baka, N., Booij, O., and Hofmann, M. (2018). El-gan: Embedding loss driven generative adversarial networks for lane detection. European Conference on Computer Vision (ECCV), Springer.10.1007/978-3-030-11009-3_15 ↗
97. He, X., Duan, Z., Chen, C., and You, F. (2019, January 6–8). Video-based lane detection and tracking during night. Proceedings of the 19th COTA International Conference of Transportation Professionals, Nanjing, China.10.1061/9780784482292.498 ↗
98. Neven, D., De Brabandere, B., Georgoulis, S., Proesmans, M., and Van Gool, L. (2018, January 26–30). Towards end-to-end lane detection: An instance segmentation approach. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.10.1109/IVS.2018.8500547 ↗
99. Fast learning method for convolutional neural networks using extreme learning machine and its application to lane detection10.1016/j.neunet.2016.12.002 ↗
100. Van Gansbeke, W., De Brabandere, B., Neven, D., Proesmans, M., and Van Gool, L. (2019, January 27–28). End-to-end lane detection through differentiable least-squares fitting. Proceedings of the IEEE International Conference on Computer Vision Workshops, Seoul, Korea.10.1109/ICCVW.2019.00119 ↗