journal article·2023-02-26Open access

Recognition and Prediction of Pedestrian Hazardous Crossing Intentions in Visual Field Obstruction Areas Based on IPVO-LSTM

Jincao Zhou, Xin Bai, Wenjie Hu

Applied Sciences· Volume 13· Issue 5

Abstract

Pedestrians who suddenly cross the street from within the blind spot of a vehicle’s field of view can pose a significant threat to traffic safety. The dangerous pedestrian crossing intentions in view-obscured scenarios have not received as much attention as the prediction of pedestrian crossing intentions. In this paper, we present a method for recognizing and predicting the dangerous crossing intention of pedestrians in a view-obscured region based on the interference, pose, velocity observation–long short-term memory (IPVO-LSTM) algorithm from a road-based view. In the first step, the road-based camera captures the pedestrian’s image. Then, we construct a pedestrian interference state feature module, pedestrian three-dimensional pose feature module, pedestrian velocity feature module, and pedestrian blind observation state feature module and extract the corresponding features of the studied pedestrians. Finally, the pedestrian hazard crossing intention prediction module based on a feature-fused LSTM (ff-LSTM) and attention mechanism is used to fuse and process the above features in a cell state process to recognize and predict the pedestrian hazard crossing intention in the blind visual area. Experiments are compared with current common algorithms in terms of the input parameter selection, intention recognition algorithm, and intention prediction time range, and the experimental results validate our state-of-the-art method.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Autonomous Vehicle Technology and Safety · Traffic and Road Safety · Video Surveillance and Tracking Methods

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

College of Mechanical and Precision Instrument Engineering, Xi’an University of Technology, Xi’an 710048, China

Funding

National Natural Science Foundation of China · Natural Science Foundation of Shaanxi Province · Research Initiation Fund of Xi’an University of Technology

References (49)

Show source-provided references
1. WHO (2021, December 24). Global Status Report on Road Safety 2018 (World Health Organization (WHO). Available online: http://www.who.int/violence_injury_prevention/road_safety_status/2018/en/.
2. A Game-Theoretic Approach to Replanning-Aware Interactive Scene Prediction and Planning10.1109/TVT.2015.2508009 ↗
3. Sighencea, B.I., Stanciu, R.I., and Căleanu, C.D. (2021). A Review of Deep Learning-Based Methods for Pedestrian Trajectory Prediction. Sensors, 21.10.3390/s21227543 ↗
4. Intention Recognition of Pedestrians and Cyclists by 2D Pose Estimation10.1109/TITS.2019.2946642 ↗
5. Xue, H., Huynh, D.Q., and Reynolds, M. (2018, January 12–15). SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.10.1109/WACV.2018.00135 ↗
6. Pedestrian Crossing Intention Prediction at Red-Light Using Pose Estimation10.1109/TITS.2021.3074829 ↗
7. Holistic LSTM for Pedestrian Trajectory Prediction10.1109/TIP.2021.3058599 ↗
8. Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., and Savarese, S. (July, January 26). Social LSTM: Human Trajectory Prediction in Crowded Spaces. Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
9. Predicting Pedestrian Intention to Cross the Road10.1109/ACCESS.2020.2987777 ↗
10. HSTA: A Hierarchical Spatio-Temporal Attention Model for Trajectory Prediction10.1109/TVT.2021.3115018 ↗
11. Human Trajectory Forecasting in Crowds: A Deep Learning Perspective10.1109/TITS.2021.3069362 ↗
12. Pedestrian intention prediction: A convolutional bottom-up multi-task approach10.1016/j.trc.2021.103259 ↗
13. Chen, T., Tian, R., and Ding, Z. (2021, January 11–17). Visual Reasoning using Graph Convolutional Networks for Predicting Pedestrian Crossing Intention. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00345 ↗
14. Yau, T., Malekmohammadi, S., Rasouli, A., Lakner, P., Rohani, M., and Luo, J. (June, January 30). Graph-SIM: A Graph-based Spatiotemporal Interaction Modelling for Pedestrian Action Prediction. Proceedings of the 2021 IEEE International Conference on Robotics and Automation, Xi’an, China.10.1109/ICRA48506.2021.9561107 ↗
15. Sun, J., Jiang, Q., and Lu, C. (2020, January 14–19). Recursive Social Behavior Graph for Trajectory Prediction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00074 ↗
16. Ma, W.-C., Huang, D.-A., Lee, N., and Kitani, K.M. (2017, January 21–26). Forecasting Interactive Dynamics of Pedestrians with Fictitious Play. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Venice, Italy.10.1109/CVPR.2017.493 ↗
17. Saleh, K., Hossny, M., and Nahavandi, S. (2019, January 20–24). Real-time Intent Prediction of Pedestrians for Autonomous Ground Vehicles via Spatio-Temporal DenseNet. Proceedings of the 2019 International Conference on Robotics and Automation, Montreal, QC, Canada.10.1109/ICRA.2019.8793991 ↗
18. Intentions of Vulnerable Road Users—Detection and Forecasting by Means of Machine Learning10.1109/TITS.2019.2923319 ↗
19. Humanlike Behavior Generation in Urban Environment Based on Learning-Based Potentials with a Low-Cost Lane Graph10.1109/TIV.2017.2788194 ↗
20. Probabilistic Prediction of Pedestrian Crossing Intention Using Roadside LiDAR Data10.1109/ACCESS.2019.2927889 ↗
21. What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction10.1109/LRA.2020.2969925 ↗
22. A Deep Learning Approach to Detect Real-Time Vehicle Maneuvers Based on Smartphone Sensors10.1109/TITS.2020.3032055 ↗
23. Konrad, S.G., Shan, M., Masson, F.R., Worrall, S., and Nebot, E. (2018, January 26–30). Pedestrian Dynamic and Kinematic Information Obtained from Vision Sensors. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.10.1109/IVS.2018.8500527 ↗
24. Yan, Z., Duckett, T., and Bellotto, N. (2017, January 24–28). Online learning for human classification in 3D LiDAR-based tracking. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202247 ↗
25. Rasouli, A., Kotseruba, I., Kunic, T., and Tsotsos, J. (November, January 27). PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
26. Kotseruba, I., Rasouli, A., and Tsotsos, J.K. (2016). Joint attention in autonomous driving (JAAD). arXiv.
27. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
28. Person Tracking in Large Public Spaces Using 3-D Range Sensors10.1109/THMS.2013.2283945 ↗
29. Strigel, E., Meissner, D., Seeliger, F., Wilking, B., and Dietmayer, K. (2014, January 8–11). The Ko-PER intersection laserscanner and video dataset. Proceedings of the 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), Qingdao, China.10.1109/ITSC.2014.6957976 ↗
30. Robicquet, A., Sadeghian, A., Alahi, A., and Savarese, S. (2016, January 11–14). Learning Social Etiquette: Human Trajectory Understanding in Crowded Scenes. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_33 ↗
31. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The Cityscapes Dataset for Semantic Urban Scene Understanding. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
32. Awad, G., Butt, A.A., Curtis, K., Lee, Y., Fiscus, J., Godil, A., and Blasi, S. (2018, January 27). Trecvid 2018: Benchmarking video activity detection, video captioning and matching, video storytelling linking and video search. Proceedings of TRECVID, Gaithersburg, MD, USA.
33. The ApolloScape Open Dataset for Autonomous Driving and Its Application10.1109/TPAMI.2019.2926463 ↗
34. Bock, J., Krajewski, R., Moers, T., Runde, S., Vater, L., and Eckstein, L. (November, January 19). The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections. Proceedings of the 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA.10.1109/IV47402.2020.9304839 ↗
35. Chang, M.-F., Ramanan, D., Hays, J., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., and Carr, P. (2019, January 15–20). Argoverse: 3D Tracking and Forecasting with Rich Maps. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00895 ↗
36. Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13–19). nuScenes: A Multimodal Dataset for Autonomous Driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01164 ↗
37. Kesten, R., Usman, M., Houston, J., Pandya, T., Nadhamuni, K., Ferreira, A., Yuan, M., Low, B., Jain, A., and Ondruska, S. (2019, September 01). Lyft Level 5 av Dataset. Available online: https://level5.lyft.com/dataset.
38. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., and Caine, B. (2020, January 14–19). Scalability in Perception for Autonomous Driving: Waymo Open Dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00252 ↗
39. Patil, A., Malla, S., Gang, H., and Chen, Y.-T. (2019, January 20–24). The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793925 ↗
40. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
41. Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition10.1186/s13640-019-0476-x ↗
42. Harris, D., and Harris, S. (2015). Digital Design and Computer Architecture, Morgan Kaufmann. [2nd ed.].10.1016/B978-0-12-800056-4.00006-6 ↗
43. Arthur, D., and Vassilvitskii, S. (2007). K-Means++: The Advantages of Careful Seeding, Stanford University.
44. Shahroudy, A., Liu, J., Ng, T.T., and Wang, G. (2016, January 27–30). NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.115 ↗
45. Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates10.1109/TPAMI.2017.2771306 ↗
46. Pedestrian Motion Trajectory Prediction with Stereo-Based 3D Deep Pose Estimation and Trajectory Learning10.1109/ACCESS.2020.2969994 ↗
47. Zhang, Y., Li, W., Wang, X., Zhao, J., and Yuan, Y. (2010, January 25–27). General design of vehicle safety assistant system based on machine vision and electronic control steering. Proceedings of the 2010 International Conference on Computer Design and Applications, Qinhuangdao, China.
48. Han, M., Zhang, D.J., Wang, Y., Yan, R., Yao, L., Chang, X., and Qiao, Y. (2022, January 18–24). Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00300 ↗
49. Quantum decision making in automatic driving10.1038/s41598-022-14737-2 ↗