Towards Explainable Pedestrian Behavior Prediction: A Neuro-Symbolic Framework for Autonomous Driving
Source abstract
Abstract
In the context of autonomous driving, predicting pedestrian behavior is a critical component for enhancing road safety. Currently, the focus of such predictions extends beyond accuracy and reliability, placing increasing emphasis on the explainability and interpretability of the models. This research presents a novel neuro-symbolic approach that integrates deep learning with fuzzy logic to develop a pedestrian behavior predictor. The proposed model leverages a set of explainable features and utilizes a fuzzy inference system to determine whether a pedestrian is likely to cross the street. The pipeline was trained and evaluated using both the Pedestrian Intention Estimation (PIE) and Joint Attention for Autonomous Driving (JAAD) datasets. The results provide experimental insights into achieving greater explainability in pedestrian behavior prediction. Additionally, the proposed method was applied to assess the data selection process through a series of experiments, leading to a set of guidelines and recommendations for data selection, feature engineering, and explainability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Autonomous Vehicle Technology and Safety · Anomaly Detection Techniques and Applications · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Engineering Department, University of Alcalá, Ctra. Madrid-Barcelona km. 33, 28805 Alcalá de Henares, Spain
Funding
HEIDI project of the European Commission · SEGVAUTO5G-CM project of the Regional Government of Madrid
References (37)
Show source-provided references
1. Slootmans, F. (2022). European Road Safety Observatory, European Commission. Technical Report.
2. World Health Organization (2023). Global Status Report on Road Safety 2023, World Health Organization.
3. Kotseruba, I., Rasouli, A., and Tsotsos, J.K. (November, January 19). Do They Want to Cross? Understanding Pedestrian Intention for Behavior Prediction. Proceedings of the 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA.10.1109/IV47402.2020.9304591 ↗
4. Cadena, P.R.G., Yang, M., Qian, Y., and Wang, C. (2019, January 27–30). Pedestrian Graph: Pedestrian Crossing Prediction Based on 2D Pose Estimation and Graph Convolutional Networks. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.10.1109/ITSC.2019.8917118 ↗
5. Lorenzo, J., Alonso, I.P., Izquierdo, R., Ballardini, A.L., Saz, A.H., Llorca, D.F., and Sotelo, M.A. (2021). Capformer: Pedestrian Crossing Action Prediction Using Transformer. Sensors, 21.10.3390/s21175694 ↗
6. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015). Learning Spatiotemporal Features with 3D Convolutional Networks. arXiv.10.1109/ICCV.2015.510 ↗
7. Rasouli, A., Kotseruba, I., and Tsotsos, J.K. (2020). Pedestrian Action Anticipation Using Contextual Feature Fusion in Stacked RNNs. arXiv.
8. Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., and Woo, W.-C. (2015). Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. Adv. Neural Inf. Process. Syst., 28, Available online: https://proceedings.neurips.cc/paper_files/paper/2015/file/07563a3fe3bbe7e3ba84431ad9d055af-Paper.pdf.
9. Kotseruba, I., Rasouli, A., and Tsotsos, J.K. (2021, January 3–8). Benchmark for Evaluating Pedestrian Action Prediction. Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV48630.2021.00130 ↗
10. Achaji, L., Moreau, J., Aioun, F., and Charpillet, F. (2023). Analysis Over Vision-Based Models for Pedestrian Action Anticipation. arXiv.10.1109/ITSC57777.2023.10422283 ↗
11. Muscholl, N., Klusch, M., Gebhard, P., and Schneeberger, T. (2021, January 22–26). Emidas: Explainable Social Interaction-Based Pedestrian Intention Detection Across Street. Proceedings of the 36th Annual ACM Symposium on Applied Computing, Virtual.10.1145/3412841.3441891 ↗
12. Marcinkevičs, R., and Vogt, J.E. (2023). Interpretability and Explainability: A Machine Learning Zoo Mini-Tour. arXiv.
13. Lipton, Z.C. (2017). The Mythos of Model Interpretability. arXiv.
14. Molnar, C. (2024, July 18). Interpretable Machine Learning. Available online: https://christophm.github.io/interpretable-ml-book.
15. Atakishiyev, S., Salameh, M., Yao, H., and Goebel, R. (2023). Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions. arXiv.10.1109/ACCESS.2024.3431437 ↗
16. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., Daumé, H., and Crawford, K. (2018). Datasheets for Datasets. arXiv.
17. Bhardwaj, A.P., Bhattacherjee, S., Chavan, A., Deshpande, A., Elmore, A.J., Madden, S., and Parameswaran, A.G. (2015, January 4–7). Datahub: Collaborative Data Science & Dataset Version Management at Scale. Proceedings of the Seventh Biennial Conference on Innovative Data Systems Research, CIDR 2015, Asilomar, CA, USA. Available online: http://cidrdb.org/cidr2015/Papers/CIDR15_Paper18.pdf.
18. Provenance in Databases: Why, How, and Where10.1561/1900000006 ↗
19. Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. (2020, January 30). Selection via Proxy: Efficient Data Selection for Deep Learning. Proceedings of the International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. Available online: https://openreview.net/forum?id=HJg2b0VYDr.
20. Malla, S., Dariush, B., and Choi, C. (2020, January 13–19). Titan: Future Forecast Using Action Priors. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01120 ↗
21. Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction10.1109/LRA.2020.2976305 ↗
22. Rasouli, A., Kotseruba, I., and Tsotsos, J.K. (2017, January 22–29). Are They Going to Cross? A Benchmark Dataset and Baseline for Pedestrian Crosswalk Behavior. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV 2017), Venice, Italy.10.1109/ICCVW.2017.33 ↗
23. Rasouli, A., Kotseruba, I., Kunic, T., and Tsotsos, J. (November, January 27). PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00636 ↗
24. Rule Weight Specification in Fuzzy Rule-Based Classification Systems10.1109/TFUZZ.2004.841738 ↗
25. GP-COACH: Genetic Programming Based Learning of Compact and Accurate Fuzzy Rule Based Classification Systems for High Dimensional Problems10.1016/j.ins.2009.12.020 ↗
26. IVTURS: A Linguistic Fuzzy Rule-Based Classification System Based on a New Interval-Valued Fuzzy Reasoning Method with Tuning and Rule Selection10.1109/TFUZZ.2013.2243153 ↗
27. A Fuzzy Association Rule-Based Classification Model for High-Dimensional Problems with Genetic Rule Selection and Lateral Tuning10.1109/TFUZZ.2011.2147794 ↗
28. Effect of Rule Weights in Fuzzy Rule-Based Classification Systems10.1109/91.940964 ↗
29. Fuzzy Identification of Systems and Its Applications to Modeling and Control10.1109/TSMC.1985.6313399 ↗
30. Burgermeister, D., and Curio, C. (2022, January 4–9). PedRecNet: Multi-Task Deep Neural Network for Full 3D Human Pose and Orientation Estimation. Proceedings of the 2022 IEEE Intelligent Vehicles Symposium (IV), Aachen, Germany.10.1109/IV51971.2022.9827202 ↗
31. Testing Predictive Automated Driving Systems: Lessons Learned and Future Recommendations10.1109/MITS.2022.3170649 ↗
32. Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition10.1016/j.patcog.2021.108487 ↗
33. Cao, Z., Simon, T., Wei, S.-E., and Sheikh, Y. (2017, January 21–26). Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
34. Papandreou, G., Zhu, T., Chen, L.-C., Gidaris, S., Tompson, J., and Murphy, K. (2018, January 8–14). PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_17 ↗
35. Han, C., Zhao, Q., Zhang, S., Chen, Y., Zhang, Z., and Yuan, J. (2022). YOLOPv2: Better, Faster, Stronger for Panoptic Driving Perception. arXiv.
36. KEEL 3.0: An Open Source Software for Multi-Stage Analysis in Data Mining10.2991/ijcis.10.1.82 ↗
37. Simpful: A User-Friendly Python Library for Fuzzy Logic10.2991/ijcis.d.201012.002 ↗