Pedestrian and Cyclist Detection and Intent Estimation for Autonomous Vehicles: A Survey
Source abstract
Abstract
As autonomous vehicles become more common on the roads, their advancement draws on safety concerns for vulnerable road users, such as pedestrians and cyclists. This paper presents a review of recent developments in pedestrian and cyclist detection and intent estimation to increase the safety of autonomous vehicles, for both the driver and other road users. Understanding the intentions of the pedestrian/cyclist enables the self-driving vehicle to take actions to avoid incidents. To make this possible, development of methods/techniques, such as deep learning (DL), for the autonomous vehicle will be explored. For example, the development of pedestrian detection has been significantly advanced using DL approaches, such as; Fast Region-Convolutional Neural Network (R-CNN) , Faster R-CNN and Single Shot Detector (SSD). Although DL has been around for several decades, the hardware to realise the techniques have only recently become viable. Using these DL methods for pedestrian and cyclist detection and applying it for the tracking, motion modelling and pose estimation can allow for a successful and accurate method of intent estimation for the vulnerable road users. Although there has been a growth in research surrounding the study of pedestrian detection using vision-based approaches, further attention should include focus on cyclist detection. To further improve safety for these vulnerable road users (VRUs), approaches such as sensor fusion and intent estimation should be investigated.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Autonomous Vehicle Technology and Safety · Advanced Neural Network Applications · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computing, Electronics and Mathematics, Coventry University, Coventry CV1 5FB, UK
School of Mechanical, Aerospace and Automotive Engineering, Coventry University, Coventry CV1 5FB, UK
Funding
Coventry University
References (100+)
Show source-provided references
1. Pedestrian Movement Direction Recognition Using Convolutional Neural Networks10.1109/TITS.2017.2726140 ↗
2. Prabhakar, G., Kailath, B., Natarajan, S., and Kumar, R. (2017, January 14–16). Obstacle detection and classification using deep learning for tracking in high-speed autonomous driving. Proceedings of the 2017 IEEE Region 10 Symposium (TENSYMP), Cochin, India.10.1109/TENCONSpring.2017.8069972 ↗
3. Tumas, P., Jonkus, A., and Serackis, A. (2018, January 26). Acceleration of HOG based Pedestrian Detection in FIR Camera Video Stream. Proceedings of the 2018 Open Conference of Electrical, Electronic and Information Sciences (eStream), Vilnius, Lithuania.10.1109/eStream.2018.8394126 ↗
4. Savasturk, D., Froehlich, B., Schneider, N., Enzweiler, M., and Franke, U. (2015, January 15–18). A Comparison Study on Vehicle Detection in Far Infrared and Regular Images. Proceedings of the IEEE Conference on Intelligent Transportation Systems, ITSC, Las Palmas, Spain.10.1109/ITSC.2015.260 ↗
5. Ren, J., Chen, X., Liu, J., Sun, W., Pang, J., Yan, Q., Tai, Y.W., and Xu, L. (2017, January 21–26). Accurate Single Stage Detector Using Recurrent Rolling Convolution. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.87 ↗
6. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
7. Szegedy, C., Liu, W., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2014, January 24–27). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2015.7298594 ↗
8. Pushing the Limits of Deep CNNs for Pedestrian Detection10.1109/TCSVT.2017.2648850 ↗
9. Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
10. Hariharan, B., Arbeláez, P., Girshick, R., and Malik, J. (2014). Simultaneous Detection and Segmentation. European Conference on Computer Vision, Springer.10.1007/978-3-319-10584-0_20 ↗
11. An Evaluation of the State of the Art10.1109/TPAMI.2011.155 ↗
12. World Health Organisation (2015). Global Status Report on Road Safety 2015—Summary, WHO.
13. Supporting a decade of action
14. Bieshaar, M., Reitberger, G., Zernetsch, S., Sick, B., Fuchs, E., and Doll, K. (2018). Detecting Intentions of Vulnerable Road Users Based on Collective Intelligence. arXiv.
15. European Commission (2017). 2017 Road Safety Statistics: What Is behind the Figures?, European Commission.
16. Robineau, D. (2017). Reported Road Casualties Great Britain, Annual Report 2017, Department for Transport.
17. Baek, J., Hong, S., Kim, J., and Kim, E. (2017). Efficient pedestrian detection at nighttime using a thermal camera. Sensors, 17.10.3390/s17081850 ↗
18. European Road Safety Observatory (2012). Traffic Safety Basic Facts 2012, European Road Safety Observatory. Technical Report.
19. Sun, W., Zhu, S., Ju, X., and Wang, D. (2018, January 9–11). Deep learning based pedestrian detection. Proceedings of the Chinese Control And Decision Conference (CCDC), Shenyang, China.10.1109/CCDC.2018.8407277 ↗
20. Multi-cue Pedestrian Detection and Tracking from a Moving Vehicle10.1007/s11263-006-9038-7 ↗
21. Gerónimo, D., Sappa, A.D., López, A., and Ponsa, D. (2007, January 21–24). Adaptive Image Sampling and Windows Classification for On—board Pedestrian Detection. Proceedings of the International Conference on Computer Vision Systems, Bielefeld, Germany.
22. Shashua, A., Gdalyahu, Y., and Hayun, G. (2004, January 14–17). Pedestrian detection for driving assistance systems: Single-frame classification and system level performance. Proceedings of the IEEE Intelligent Vehicles Symposium, Parma, Italy.
23. Computer vision and deep learning techniques for pedestrian detection and tracking: A survey10.1016/j.neucom.2018.01.092 ↗
24. Shaout, A., Colella, D.M., and Awad, S.S. (2011, January 27–28). Advanced Driver Assistance Systems—Past, present and future. Proceedings of the 2011 Seventh International Computer Engineering Conference (ICENCO’2011), Cairo, Egypt.10.1109/ICENCO.2011.6153935 ↗
25. A unified framework for concurrent pedestrian and cyclist detection10.1109/TITS.2016.2567418 ↗
26. Wang, H., Chen, Q., and Cai, W. (2006, January 4–6). Shape-based pedestrian/bicyclist detection via onboard stereo vision. Proceedings of the Multiconference on “Computational Engineering in Systems Applications”, Beijing, China.10.1109/CESA.2006.313601 ↗
27. Noyce, D.A., Dharmaraju, R., and Lehman, J.D. (2002). An Evaluation of Technologies for Automated Detection and Classification of Pedestrians and Bicyclists, Massachusetts Highway Department Report.
28. Solichin, A., Harjoko, A., and Eko, A. (2014). A Survey of Pedestrian Detection in Video. Int. J. Adv. Comput. Sci. Appl., 5.10.14569/IJACSA.2014.051007 ↗
29. Piccard, M. (2004). Background subtraction techniques: A review. J. Hepatol.10.1109/ICSMC.2004.1400815 ↗
30. Benezeth, Y., Jodoin, P., Emile, B., Laurent, H., and Rosenberger, C. (2008, January 8–11). Review and evaluation of commonly-implemented background subtraction algorithms. Proceedings of the 2008 19th International Conference on Pattern Recognition, Tampa, FL, USA.10.1109/ICPR.2008.4760998 ↗
31. Agarwal, S., Terrail, J.O.D., and Jurie, F. (2018). Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks. arXiv.
32. Dalal, N., and Triggs, B. (2005, January 20–25). Histograms of Oriented Gradients for Human Detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
33. Multiple Sensor Fusion and Classification for Moving Object Detection and Tracking10.1109/TITS.2015.2479925 ↗
34. Wang, X., Han, T.X., and Yan, S. (October, January 29). An HOG-LBP human detector with partial occlusion handling. Proceedings of the IEEE 12th International Conference on Computer Vision, Kyoto, Japan.
35. Object Detection with Discriminatively Trained Part-Based Models10.1109/TPAMI.2009.167 ↗
36. Roncancio, H., Hernandes, A.C., and Becker, M. (2012, January 2–4). Vision-based system for pedestrian recognition using a tuned SVM classifier. Proceedings of the Workshop on Engineering Applications, Bogota, Columbia.10.1109/WEA.2012.6220095 ↗
37. Yang, Y., Liu, W., Wang, Y., and Cai, Y. (2012, January 19–22). Research on the algorithm of pedestrian recognition in front of the vehicle based on SVM. Proceedings of the 11th International Symposium on Distributed Computing and Applications to Business, Engineering and Science, DCABES 2012, Guilin, China.10.1109/DCABES.2012.108 ↗
38. Min, K., Son, H., Choe, Y., and Kim, Y.G. (2013, January 19–21). Real-time pedestrian detection based on A hierarchical two-stage Support Vector Machine. Proceedings of the 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA), Melbourne, Australia.
39. Neagoe, V.E., Ciotec, A.D., and Bărar, A.P. (2012, January 21–23). A Concurrent Neural Network Approach to Pedestrian Detection in Thermal Imagery. Proceedings of the 9th International Conference on Communications (COMM), Bucharest, Romania.10.1109/ICComm.2012.6262539 ↗
40. Brazil, G., Yin, X., and Liu, X. (2017, January 22–29). Illuminating Pedestrians via Simultaneous Detection and Segmentation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.530 ↗
41. Oren, M., Papageorgiou, C., Sinha, P., Osuna, E., and Poggio, T. (1997, January 17–19). Pedestrian detection using wavelet templates. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, PR, USA.
42. Viola, P., Jones, M.J., and Snow, D. (2003, January 13–16). Detecting Pedestrians Using Patterns of Motion and Appearance. Proceedings of the 9th IEEE International Conference on Computer Vision, Nice, France.10.1109/ICCV.2003.1238422 ↗
43. Benenson, R., Omran, M., Hosang, J., and Schiele, B. (2015). Ten Years of Pedestrian Detection, What Have We Learned? Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer.10.1007/978-3-319-16181-5_47 ↗
44. Deep Convolutional Neural Networks for pedestrian detection10.1016/j.image.2016.05.007 ↗
45. A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting10.1006/jcss.1997.1504 ↗
46. Felzenszwalb, P., McAllester, D., and Ramanan, D. (2008, January 23–28). A discriminatively trained, multiscale, deformable part model. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.10.1109/CVPR.2008.4587597 ↗
47. Cho, H., Rybski, P.E., Bar-Hillel, A., and Zhang, W. (2012, January 3–7). Real-time pedestrian detection with deformable part models. Proceedings of the 2012 IEEE Intelligent Vehicles Symposium, Alcala de Henares, Spain.10.1109/IVS.2012.6232264 ↗
48. Li, X., Flohr, F., Yang, Y., Xiong, H., Braun, M., Pan, S., Li, K., Gavrila, D.M., and Flohr, F. (2016, January 19–22). A new benchmark for vision-based cyclist detection. Proceedings of the IEEE Intelligent Vehicles Symposium, Gotenburg, Sweden.
49. Object detection with discriminatively trained part-based models10.1109/MC.2014.42 ↗
50. Ghosh, S., Amon, P., Hutter, A., and Kaup, A. (2017, January 17–20). Reliable pedestrian detection using a deep neural network trained on pedestrian counts. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.10.1109/ICIP.2017.8296368 ↗
51. Region-based convolutional networks for accurate object detection and segmentation10.1109/TPAMI.2015.2437384 ↗
52. Cho, H., Rybski, P.E., and Zhang, W. (2010, January 21–24). Vision-based bicyclist detection and tracking for intelligent vehicles. Proceedings of the 2010 IEEE Intelligent Vehicles Symposium, San Diego, CA, USA.10.1109/IVS.2010.5548063 ↗
53. Tian, W., and Lauer, M. (2015, January 11–14). Fast and Robust Cyclist Detection for Monocular Camera Systems. Proceedings of the International joint Conference on Computer Vision Imaging and Computer Graphics Theory and Applications (VISIGRAPP), Berlin, Germany.
54. Tong, L., Cao, X., and Yanwu, X. (2010, January 7–9). An effective crossing cyclist detection on a moving vehicle. Proceedings of the 2010 8th World Congress on Intelligent Control and Automation, Jinan, China.10.1109/WCICA.2010.5554979 ↗
55. Mitter, C.S. (Sensors Magazine, 2017). Autonomous Car Ingredients: Safety, Surveillance and Infotainment, Part Three, Sensors Magazine.
56. (2018). JdeRobot_DetectionSuite, GitHub Inc.. ROI_HOG.
57. Moving Object Localization Using Optical Flow for Pedestrian Detection from a Moving Vehicle10.1155/2014/196415 ↗
58. Mukherjee, R. (2018). Classification.
59. Ren, S., He, K., Girshick, R., and Sun, J. (2017). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell., 39.10.1109/TPAMI.2016.2577031 ↗
60. Selective search for object recognition10.1007/s11263-013-0620-5 ↗
61. Nam, W., Dollár, P., and Han, J.H. (2014). Local Decorrelation For Improved Detection. arXiv.
62. Human detection from images and videos: A survey10.1016/j.patcog.2015.08.027 ↗
63. Taigman, Y., Yang, M., Ranzato, M., and Wolf, L. (2014, January 23–28). DeepFace: Closing the gap to human-level performance in face verification. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.220 ↗
64. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Annual Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
65. Ordóñez, F.J., and Roggen, D. (2016). Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors, 16.10.3390/s16010115 ↗
66. Palaz, D., Doss, M.M., and Collobert, R. (2015, January 6–10). Analysis of CNN-based Speech Recognition System using Raw Speech as Input. Proceedings of the Conference of International Speech Communication Association (Interspeech), Dresden, Germang.10.21437/Interspeech.2015-3 ↗
67. Bo Yang, J., Nhut Nguyen, M., Phyo San, P., Li Li, X., and Krishnaswamy, S. (2015, January 25–31). Deep Convolutional Neural Networks On Multichannel Time Series For Human Activity Recognition. Proceedings of the International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina.
68. Dollár, P., Babenko, B., Belongie, S., Perona, P., and Tu, Z. (2008, January 12–18). Multiple Component Learning for Object Detection. Proceedings of the ECCV, Marseille, France.10.1007/978-3-540-88688-4_16 ↗
69. Ess, A., Leibe, B., Schindler, K., and Van Gool, L. (2008, January 23–28). A mobile vision system for robust multi-person tracking. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.10.1109/CVPR.2008.4587581 ↗
70. Lin, Z., and Davis, L.S. (2008, January 12–18). A Pose-Invariant Descriptor for Human Detection and Segmentation. Proceedings of the Computer Vision—ECCV 2008, Marseille, France.10.1007/978-3-540-88693-8_31 ↗
71. Pedestrian Detection and Tracking Using a Mixture of View-Based Shape–Texture Models10.1109/TITS.2008.922943 ↗
72. Sabzmeydani, P., and Mori, G. (2007, January 17–22). Detecting Pedestrians by Learning Shapelet Features. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.10.1109/CVPR.2007.383134 ↗
73. Seemann, E., Fritz, M., and Schiele, B. (2007, January 17–22). Towards Robust Pedestrian Detection in Crowded Image Sequences. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.10.1109/CVPR.2007.383300 ↗
74. Sharma, V., and Davis, J.W. (2007, January 14–21). Integrating Appearance and Motion Cues for Simultaneous Detection and Segmentation of Pedestrians. Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil.10.1109/ICCV.2007.4409035 ↗
75. Pedestrian Detection: The State of the Art10.1109/TPAMI.2011.155 ↗
76. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv.
77. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014). Caffe: Convolutional Architecture for Fast Feature Embedding. ACM, 675–678.10.1145/2647868.2654889 ↗
78. Lecun, Y., Bengio, Y., and Hinton, G. (2015). Deep Learning. Nature.10.1038/nature14539 ↗
79. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 24–27). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the EEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
80. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and Lecun, Y. (2013, January 2–4). OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks. Proceedings of the International Conference on Learning Representations, Scottsdale, Arizona.
81. Toshev, A., and Christian Szegedy, G. (2014, January 24–27). DeepPose: Human Pose Estimation via Deep Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.214 ↗
82. Sainath, T.N., Mohamed, A., Kingsbury, B., and Ramabhadran, B. (2015). Deep Convolutional Neural Networks for LVCSR. Scand. J. Rheumatol., 39–48.
83. Dai, J., Li, Y., He, K., and Sun, J. (2016, January 5–10). R-FCN: Object Detection via Region-based Fully Convolutional Networks. Proceedings of the IEEE conference on Advances in Neural Information Processing, Barcelona, Spain.
84. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016). SSD: Single Shot MultiBox Detector. European Conference on Computer Vision, Springer.10.1007/978-3-319-46448-0_2 ↗
85. Song, H., Choi, I.K., Ko, M.S., Bae, J., Kwak, S., and Yoo, J. (2018, January 24–27). Vulnerable pedestrian detection and tracking using deep learning. Proceedings of the 2018 International Conference on Electronics, Information, and Communication (ICEIC), Honolulu, HI, USA.10.23919/ELINFOCOM.2018.8330547 ↗
86. Hou, Y.L., Song, Y., Hao, X., Shen, Y., and Qian, M. (2017, January 22–25). Multispectral pedestrian detection based on deep convolutional neural networks. Proceedings of the IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), Xiamen, China.10.1109/ICSPCC.2017.8242507 ↗
87. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2015). You Only Look Once: Unified, Real-Time Object Detection. arXiv.
88. Angelova, A., Krizhevsky, A., and Vanhoucke, V. (2015, January 26–30). Pedestrian detection with a Large-Field-Of-View deep network. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.10.1109/ICRA.2015.7139256 ↗
89. Tian, Y., Luo, P., Wang, X., and Tang, X. (2015, January 7–13). Deep learning strong parts for pedestrian detection. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.221 ↗
90. Cai, Z., Fan, Q., Feris, R.S., and Vasconcelos, N. (2016). A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection. European Conference on Computer Vision, Springer.10.1007/978-3-319-46493-0_22 ↗
91. Wang, L., Xu, L., and Yang, M.H. (2016, January 25–28). Pedestrian detection in crowded scenes via scale and occlusion analysis. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.10.1109/ICIP.2016.7532550 ↗
92. Du, X., El-Khamy, M., Lee, J., and Davis, L. (2017, January 24–31). Fused DNN: A deep neural network fusion approach to fast and robust pedestrian detection. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision, WACV 2017, Santa Rosa, CA, USA.10.1109/WACV.2017.111 ↗
93. Xiang, Y., Choi, W., Lin, Y., and Savarese, S. (2017, January 24–31). Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.108 ↗
94. Yang, F., Choi, W., and Lin, Y. (2016, January 27–30). Exploit All the Layers: Fast and Accurate CNN Object Detector with Scale Dependent Pooling and Cascaded Rejection Classifiers. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.234 ↗
95. Girshick, R. (2015, January 7–13). Fast R-CNN. Proceedings of the IEEE International Conference on Computer Vision(ICCV), Santiago, Chile.10.1109/ICCV.2015.169 ↗
96. He, K., Zhang, X., Ren, S., and Sun, J. (2014). Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. European Conference on Computer Vision, Springer.10.1007/978-3-319-10578-9_23 ↗
97. González, A., Fang, Z., Socarras, Y., Serrat, J., Vázquez, D., Xu, J., and López, A.M. (2016). Pedestrian Detection at Day/Night Time with Visible and FIR Cameras: A Comparison. Sensors, 16.10.3390/s16060820 ↗
98. Erhan, D., Szegedy, C., Toshev, A., and Anguelov, D. (2014, January 23–28). Scalable Object Detection Using Deep Neural Networks. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.276 ↗
99. Szegedy, C., Reed, S., Erhan, D., Anguelov, D., and Ioffe, S. (2015). Scalable, High-Quality Object Detection. arXiv.
100. What Makes for Effective Detection Proposals?10.1109/TPAMI.2015.2465908 ↗