Risk and Complexity Assessment of Autonomous Vehicle Testing Scenarios
Source abstract
Abstract
Autonomous vehicles (AVs) must fulfill adequate safety requirements before formal application, and performing an effective functional evaluation to verify vehicle safety requires extensive testing in different scenarios. However, it is crucial to rationalize the application of different scenarios to support different testing needs; thus, one of the current challenges limiting the development of AVs is the critical evaluation of scenarios, i.e., the lack of quantitative criteria for scenario design. This study introduces a method using the Spherical Fuzzy-Analytical Network Process (SF-ANP) to evaluate these scenarios, addressing their inherent risks and complexities. The method involves constructing a five-layer model to decompose scenario elements and using SF-ANP to calculate weights based on element interactions. The study evaluates 700 scenarios from the China In-depth Traffic Safety Study–Traffic Accident (CIMSS-TA) database, incorporating fuzzy factors and element weights. Virtual simulation of vehicles in the scenarios was performed using Baidu Apollo, and the performance of the scenarios was assessed by collecting the vehicle test results. The correlation between the obtained alternative safety indicators and the quantitative values confirms the validity and scientific validity of this approach. This will provide valuable guidance for categorizing audiovisual test scenarios and selecting corresponding scenarios to challenge different levels of vehicle functionality. At the same time, it can be used as a design basis to generate a large number of effective scenarios to accelerate the construction of scenario libraries and promote commercialization of AVs.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Safety Systems Engineering in Autonomy · Risk and Safety Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Traffic & Transportation Engineering, Central South University, Changsha 410075, China
Department of Industrial and Manufacturing Systems Engineering, The University of Hong Kong, Hong Kong 999077, China
Funding
Natural Science Foundation of Hunan Province, China
References (49)
Show source-provided references
1. Ben Abdessalem, R., Nejati, S., Briand, L.C., and Stifter, T. (2016, January 3–7). Testing advanced driver assistance systems using multi-objective search and neural networks. Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, Singapore.10.1145/2970276.2970311 ↗
2. A method for predicting crash configurations using counterfactual simulations and real-world data10.1016/j.aap.2020.105932 ↗
3. Elrofai, H., Paardekooper, J.P., de Gelder, E., Kalisvaart, S., and den Camp, O.O. (2018). Scenario-Based Safety Validation of Connected and Automated Driving, TNO, Technical Report, Netherlands Organization for Applied Scientific Research.
4. Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?10.1016/j.tra.2016.09.010 ↗
5. Waymo simulated driving behavior in reconstructed fatal crashes within an autonomous vehicle operating domain10.1016/j.aap.2021.106454 ↗
6. A survey on safety-critical driving scenario generation—A methodological perspective10.1109/TITS.2023.3259322 ↗
7. Ulbrich, S., Menzel, T., Reschka, A., Schuldt, F., and Maurer, M. (2015, January 15–18). Defining and substantiating the terms scene, situation, and scenario for automated driving. Proceedings of the 2015 IEEE 18th International Conference on Intelligent Transportation Systems, Gran Canaria, Spain.10.1109/ITSC.2015.164 ↗
8. Wei, Z., Huang, H., Zhang, G., Zhou, R., Luo, X., Li, S., and Zhou, H. (2024). Interactive Critical Scenario Generation for Autonomous Vehicles Testing Based on In-depth Crash Data Using Reinforcement Learning. IEEE Trans. Intell. Veh., 1–12.10.1109/TIV.2024.3415961 ↗
9. Evaluating Autonomous Vehicle Safety Performance Through Analysis of Pre-Crash Trajectories of Powered Two-Wheelers10.1109/TITS.2024.3392673 ↗
10. Testing scenario library generation for connected and automated vehicles, part I: Methodology10.1109/TITS.2020.2972211 ↗
11. Pre-crash scenarios at road junctions: A clustering method for car crash data10.1016/j.aap.2017.07.011 ↗
12. Survey on scenario-based safety assessment of automated vehicles10.1109/ACCESS.2020.2993730 ↗
13. Research on the quantitative evaluation of the traffic environment complexity for unmanned vehicles in urban roads10.1109/ACCESS.2021.3054882 ↗
14. Risk analysis of autonomous vehicle test scenarios using a novel analytic hierarchy process method10.1049/itr2.12466 ↗
15. Cheng, Y., Liu, Z., Gao, L., Zhao, Y., and Gao, T. (2022). Traffic risk environment impact analysis and complexity assessment of autonomous vehicles based on the potential field method. Int. J. Environ. Res. Public Health, 19.10.3390/ijerph191610337 ↗
16. Driving safety field theory modeling and its application in pre-collision warning system10.1016/j.trc.2016.10.003 ↗
17. Zhang, L., Ma, Y., Xing, X., Xiong, L., and Chen, J. (2021, January 19–22). Research on the complexity quantification method of driving scenarios based on information entropy. Proceedings of the 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA.10.1109/ITSC48978.2021.9565025 ↗
18. Schuldt, F. (2017). Towards Testing of Automated Driving Functions in Virtual Driving Environments. [Ph.D. Thesis, Technical University Brunswick].
19. Bagschik, G., Menzel, T., and Maurer, M. (2018, January 26–30). Ontology based scene creation for the development of automated vehicles. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.10.1109/IVS.2018.8500632 ↗
20. Definition of scenarios for safety validation of automated driving functions10.1007/s38311-018-0197-2 ↗
21. Thorn, E., Kimmel, S.C., Chaka, M., and Hamilton, B.A. (2018). A Framework for Automated Driving System Testable Cases and Scenarios, United States, Department of Transportation, National Highway Traffic Safety Administration. Technical Report.
22. Survey of deep reinforcement learning for motion planning of autonomous vehicles10.1109/TITS.2020.3024655 ↗
23. Deep reinforcement learning for autonomous driving: A survey10.1109/TITS.2021.3054625 ↗
24. Zhao, D., Guo, Y., and Jia, Y.J. (2017, January 16–19). Trafficnet: An open naturalistic driving scenario library. Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan.10.1109/ITSC.2017.8317860 ↗
25. Wang, Y., Li, K., Hu, Y., and Chen, H. (2020, January 22–24). Modeling and quantitative assessment of environment complexity for autonomous vehicles. Proceedings of the 2020 Chinese Control And Decision Conference (CCDC), Hefei, China.10.1109/CCDC49329.2020.9164239 ↗
26. Dynamic driving environment complexity quantification method and its verification10.1016/j.trc.2021.103051 ↗
27. Wang, J., Zhang, C., Liu, Y., and Zhang, Q. (2018, January 26–30). Traffic sensory data classification by quantifying scenario complexity. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.10.1109/IVS.2018.8500669 ↗
28. Rudnik, K., Chwastyk, A., and Pisz, I. (2024). Approach Based on the Ordered Fuzzy Decision Making System Dedicated to Supplier Evaluation in Supply Chain Management. Entropy, 26.10.3390/e26100860 ↗
29. Bulut, M.S., Ordu, M., Der, O., and Basar, G. (2024). Sustainable Thermoplastic Material Selection for Hybrid Vehicle Battery Packs in the Automotive Industry: A Comparative Multi-Criteria Decision-Making Approach. Polymers, 16.10.3390/polym16192768 ↗
30. Saaty, T.L. (1996). Decision Making with Dependence and Feedback: The Analytic Network Process, RWS Publications.
31. Fuzzy sets10.1016/S0019-9958(65)90241-X ↗
32. The concept of a linguistic variable and its application to approximate reasoning—I10.1016/0020-0255(75)90036-5 ↗
33. Atanassov, K.T., and Atanassov, K.T. (1999). Intuitionistic Fuzzy Sets, Springer.10.1007/978-3-7908-1870-3 ↗
34. Yager, R.R. (2013, January 24–28). Pythagorean fuzzy subsets. Proceedings of the 2013 Joint IFSA World Congress and NAFIPS Annual Meeting (IFSA/NAFIPS), Edmonton, AB, Canada.10.1109/IFSA-NAFIPS.2013.6608375 ↗
35. Neutrosophic logic-a generalization of the intuitionistic fuzzy logic
36. Hesitant fuzzy sets
37. Spherical fuzzy sets and spherical fuzzy TOPSIS method10.3233/JIFS-181401 ↗
38. Evaluation of autonomous vehicle driving systems for risk assessment based on three-dimensional uncertain linguistic variables10.1016/j.asoc.2021.107934 ↗
39. Risk assessment of water supply network operation based on ANP-fuzzy comprehensive evaluation method10.1061/(ASCE)PS.1949-1204.0000602 ↗
40. Huang, J.J., and Chen, C.Y. (2023). Using Markov Random Field and Analytic Hierarchy Process to Account for Interdependent Criteria. Algorithms, 17.10.3390/a17010001 ↗
41. Yazo-Cabuya, E.J., Ibeas, A., and Herrera-Cuartas, J.A. (2024). Integration of Sustainability in Risk Management and Operational Excellence through the VIKOR Method Considering Comparisons between Multi-Criteria Decision-Making Methods. Sustainability, 16.10.3390/su16114585 ↗
42. Zhang, C., Huang, Y., Zhou, D., Dong, Z., He, S., and Zhou, Z. (2024). A MCDM-Based Analysis Method of Testability Allocation for Multi-Functional Integrated RF System. Electronics, 13.10.3390/electronics13183618 ↗
43. A novel spherical fuzzy analytic hierarchy process and its renewable energy application10.1007/s00500-019-04222-w ↗
44. The analytic hierarchy process
45. Investigating the effect of contextual factors on driving: An experimental study10.1016/j.trf.2022.05.007 ↗
46. Driving risk assessment based on naturalistic driving study and driver attitude questionnaire analysis10.1016/j.aap.2020.105680 ↗
47. Determination of takeover time budget based on analysis of driver behavior10.1109/OJITS.2022.3224677 ↗
48. Towards the unified principles for level 5 autonomous vehicles10.1016/j.eng.2020.10.018 ↗
49. Motorcyclist injury risk as a function of real-life crash speed and other contributing factors10.1016/j.aap.2018.12.010 ↗