journal article·2025-06-22Open access

Minimum Critical Test Scenario Set Selection for Autonomous Vehicles Prior to First Deployment and Public Road Testing

Balint Toth, Zsolt Szalay

Applied Sciences· Volume 15· Issue 13

Abstract

The growing complexity of autonomous vehicle functionalities poses significant challenges for vehicle testing, validation, and regulatory approval. Despite the availability of various testing protocols and standards, a harmonized and widely accepted method specifically targeting the selection of critical test scenarios—especially for safety assessments prior to public road testing—has not yet been developed. This study introduces a systematic methodology for selecting a minimum critical set of test scenarios tailored to an autonomous vehicle’s Operational Design Domain (ODD) and capabilities. Building on existing testing frameworks (e.g., EuroNCAP protocols, ISO standards, UNECE and EU regulations), the proposed method combines a structured questionnaire with a weighted cosine similarity based filtering mechanism to identify relevant scenarios from a robust database of over 1000 test cases. Further refinement using similarity metrics such as Euclidean and Manhattan distances ensures the elimination of redundant test scenarios. Application of the framework to real-world projects demonstrates significant alignment with expert-identified cases, while also identifying overlooked but relevant scenarios. By addressing the need for a structured and efficient scenario selection method, this work supports the advancement of systematic safety assurance for autonomous vehicles and provides a scalable solution for authorities and vehicle testing companies.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Autonomous Vehicle Technology and Safety · Vehicle Dynamics and Control Systems · Automotive and Human Injury Biomechanics

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Automotive Technologies, Faculty of Transportation Engineering and Vehicle Engineering, Budapest University of Technology and Economics, Műegyetem Rakpart 3, 1111 Budapest, Hungary

Funding

Ministry of Culture and Innovation of Hungary

References (32)

Show source-provided references
1. Myers, R., and Saigol, Z. (2020). Pass-Fail Criteria for Scenario-Based Testing of Automated Driving Systems. arXiv.
2. (2025, March 25). UNECE: New Assessment/Test Method for Automated Driving (NATM) Guidelines for Validating Automated Driving System (ADS). Available online: https://unece.org/sites/default/files/2022-04/ECE-TRANS-WP.29-2022-58.pdf.
3. (2025, March 25). Commission Implementing Regulation (EU) 2022/1426 of 5 August 2022 Laying down Rules for the Application of Regulation (EU) 2019/2144 of the European Parliament and of the Council as Regards Uniform Procedures and Technical Specifications for the Type-Approval of the Automated Driving System (ADS) of Fully Automated Vehicles (Text with EEA Relevance). Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32022R1426.
4. Yan, M., Rampino, L., and Caruso, G. (2025). Comparing User Acceptance in Human–Machine Interfaces Assessments of Shared Autonomous Vehicles: A Standardized Test Procedure. Appl. Sci., 15.10.3390/app15010045 ↗
5. (2025, March 25). PEGASUS Projekt: Scenario Description. Available online: https://www.pegasusprojekt.de/files/tmpl/PDF-Symposium/04_Scenario-Description.pdf.
6. (2025, February 14). UNECE 1958 Agreement: Addendum 170–UN Regulation No. 171: Uniform Provisions Concerning the Approval of Vehicles with Regard to Driver Control Assistance Systems. Available online: https://unece.org/sites/default/files/2025-01/R171e%20%282%29.pdf.
7. Vass, S., Donà, R., Mattas, K., Morandin, G., Toth, B., Heé, M.Á., Galassi, M.C., and Ciuffo, B. (2023). Are Low-Speed Automated Vehicles Ready for Deployment? Implications on Safety and Urban Traffic. Transp. Res. Part C, submitted.10.2139/ssrn.4760713 ↗
8. (2025, March 13). KBA: Autonome-Fahrzeuge-Genehmigungs-Und Betriebs-Verordnung (AFGBV). Available online: https://www.gesetze-im-internet.de/afgbv/.
9. Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?10.1016/j.tra.2016.09.010 ↗
10. Virtual Testing of Automated Driving Systems. A Survey on Validation Methods10.1109/ACCESS.2022.3153722 ↗
11. PolyVerif: An Open-Source Environment for Autonomous Vehicle Validation and Verification Research Acceleration10.1109/ACCESS.2023.3258681 ↗
12. Fields of Interest and Demands for a Digital Proving Ground Twin
13. Toth, B., and Szalay, Z. (2023). Development and Functional Validation Method of the Scenario-in-the-Loop Simulation Control Model Using Co-Simulation Techniques. Machines, 11.10.3390/machines11111028 ↗
14. Szalay, Z., Szalai, M., Tóth, B., Tettamanti, T., and Tihanyi, V. (2019, January 4–8). Proof of Concept for Scenario-in-the-Loop (SciL) Testing for Autonomous Vehicle Technology. Proceedings of the 2019 IEEE International Conference on Connected Vehicles and Expo (ICCVE), Graz, Austria.10.1109/ICCVE45908.2019.8965086 ↗
15. (2025, April 11). MATLAB Help Center: Simulate Vehicle Dynamics in ZalaZONE Automotive Proving Ground 3D Scene. Available online: https://ch.mathworks.com/help/vdynblks/ug/simulate-vehicle-dynamics-in-zalazone-automotive-proving-ground-3d-scene.html.
16. Katzorke, N., Moosmann, M., Imdahl, R., and Lasi, H. (2020, January 20–23). A Method to Assess and Compare Proving Grounds in the Context of Automated Driving Systems. Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece.10.1109/ITSC45102.2020.9294310 ↗
17. Shakeri, A. (2024, January 1–5). Formalization of Operational Domain and Operational Design Domain for Automated Vehicles. Proceedings of the 2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C), Cambridge, UK.10.1109/QRS-C63300.2024.00131 ↗
18. Eichenseer, F., Sarkar, S., and Shakeri, A. (2024, January 28–31). A Systematic Methodology for Specifying the Operational Design Domain of Automated Vehicles. Proceedings of the 2024 IEEE 35th International Symposium on Software Reliability Engineering Workshops (ISSREW), Tsukuba, Japan.10.1109/ISSREW63542.2024.00040 ↗
19. Safety Assessment of Highly Automated Driving Systems in Test Tracks: A New Framework10.1016/j.aap.2020.105664 ↗
20. Testing Scenario Library Generation for Connected and Automated Vehicles: An Adaptive Framework10.1109/TITS.2020.3023668 ↗
21. Novel Test Scenario Generation Technology for Performance Evaluation of Automated Vehicle10.1007/s12239-022-0113-z ↗
22. Wang, Z., Ma, J., and Lai, E.M.-K. (2024). A Survey of Scenario Generation for Automated Vehicle Testing and Validation. Future Internet, 16.10.3390/fi16120480 ↗
23. Wu, X., Chen, J., Xing, X., Sun, J., Tian, Y., Liu, L., and Shen, Y. (2024). LAMBDA: Covering the Multimodal Critical Scenarios for Automated Driving Systems by Search Space Quantization. arXiv.
24. Predictive Battery Thermal and Energy Management for Connected and Automated Electric Vehicles10.1109/TITS.2024.3494734 ↗
25. Reckenzaun, J., Csanady, L., Eichberger, A., Engelstein, A., Hörmann, L., Kiss, T., Luley, P., Rüther, M., Schwarz, S., and Strasser-Krauss, T. (2022, January 7–9). Transnational Testing, Operation and Certification of Automated Driving Systems: Perspective from testEPS and Central System EUREKA Projects. Proceedings of the 2022 International Conference on Connected Vehicle and Expo (ICCVE), Lakeland, FL, USA.10.1109/ICCVE52871.2022.9742896 ↗
26. Reckenzaun, J., Rott, R., Kirchengast, M., Ritter, D.J., Innerwinkler, P., Solmaz, S., Pilz, C., Schratter, M., Eichberger, A., and Mihalj, T. (2023, January 16–18). Transnational Testing, Operation and Certification of Automated Driving Systems: Perspective from testEPS and Central System EUREKA Projects-Mid-Term Results. Proceedings of the 2023 IEEE International Automated Vehicle Validation Conference (IAVVC), Austin, TX, USA.10.1109/IAVVC57316.2023.10328093 ↗
27. Reckenzaun, J., Rott, R., Nikolić, O., Innerwinkler, P., Kirchengast, M., Ritter, D.J., Solmaz, S., Ladstätter, S., Luley, P., and Toth, B. (2024, January 22–23). Transnational Testing, Operation and Certification of Automated Driving Systems–Building Bricks for the NATM Multi-Pillar Framework. Proceedings of the 2024 IEEE International Automated Vehicle Validation Conference (IAVVC), Pittsburgh, PA, USA.10.1109/IAVVC63304.2024.10786414 ↗
28. Li, J., Zong, R., Wang, Y., and Deng, W. (2024). Complexity Evaluation for Urban Intersection Scenarios in Autonomous Driving Tests: Method and Validation. Appl. Sci., 14.10.3390/app142210451 ↗
29. Learning Similarity with Cosine Similarity Ensemble10.1016/j.ins.2015.02.024 ↗
30. Steck, H., Ekanadham, C., and Kallus, N. (2024, January 13–17). Is Cosine-Similarity of Embeddings Really About Similarity?. Proceedings of the Companion ACM Web Conference 2024, Singapore.10.1145/3589335.3651526 ↗
31. Visual Tracking via Weighted Local Cosine Similarity10.1109/TCYB.2014.2360924 ↗
32. Boyd, S., and Vandenberghe, L. (2018). Chapter 3: Norm and distance. Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares, Cambridge University Press. [1st ed.].10.1017/9781108583664 ↗