Statistical Validation Framework for Automotive Vehicle Simulations Using Uncertainty Learning
Source abstract
Abstract
The modelling and simulation process in the automotive domain is transforming. Increasing system complexity and variant diversity, especially in new electric powertrain systems, lead to complex, modular simulations that depend on virtual vehicle development, testing and approval. Consequently, the emerging key requirements for automotive validation involve a precise reliability quantification across a large application domain. Validation is unable to meet these requirements because its results provide little information, uncertainties are neglected, the model reliability cannot be easily extrapolated and the resulting application domain is small. In order to address these insufficiencies, this paper develops a statistical validation framework for dynamic systems with changing parameter configurations, thus enabling a flexible validation of complex total vehicle simulations including powertrain modelling. It uses non-deterministic models to consider input uncertainties, applies uncertainty learning to predict inherent model uncertainties and enables precise reliability quantification of arbitrary system parameter configurations to form a large application domain. The paper explains the framework with real-world data from a prototype electric vehicle on a dynamometer, validates it with additional tests and compares it to conventional validation methods. It is published as an open-source document. With the validation information from the framework and the knowledge deduced from the real-world problem, the paper solves its key requirements and offers recommendations on how to efficiently revise models with the framework’s validation results.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Real-time simulation and control systems · Probabilistic and Robust Engineering Design · Vehicle Dynamics and Control Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Insitute of Automotive Technology, Technical Universiy of Munich, Boltzmannstr. 15, 85748 Garching, Germany
Funding
Bundesministerium für Bildung und Forschung
References (63)
Show source-provided references
1. Danquah, B., Riedmaier, S., and Lienkamp, M. (2020). Potential of statistical model verification, validation and uncertainty quantification in automotive vehicle dynamics simulations: A review. Veh. Syst. Dyn., 1–30.10.1080/00423114.2020.1854317 ↗
2. Whole-system thinking, development control, key barriers and promotion mechanism for EV development10.1007/s40565-015-0113-3 ↗
3. Simulation methods supporting homologation of Electronic Stability Control in vehicle variants10.1080/00423114.2017.1322705 ↗
4. Statistical Model Verification and Validation Concept in Automotive Vehicle Design10.1016/j.procir.2020.02.175 ↗
5. Nicoletti, L., Brönner, M., Danquah, B., Koch, A., Konig, A., Krapf, S., Pathak, A., Schockenhoff, F., Sethuraman, G., and Wolff, S. (2020, January 10–12). Review of Trends and Potentials in the Vehicle Concept Development Process. Proceedings of the 2020 Fifteenth International Conference on Ecological Vehicles and Renewable Energies (EVER), Monte-Carlo, Monaco.10.1109/EVER48776.2020.9243115 ↗
6. Zimmer, M. (2015). Durchgängiger Simulationsprozess zur Effizienzsteigerung und Reifegraderhöhung von Konzeptbewertungen in der frühen Phase der Produktentstehung, Springer. Wissenschaftliche Reihe Fahrzeugtechnik Universität Stuttgart.10.1007/978-3-658-11500-5 ↗
7. Scientific computer simulation review10.1016/j.ress.2015.01.020 ↗
8. Research methodology for a new validation concept in vehicle dynamics10.1007/s41104-018-0024-1 ↗
9. Tschochner, M.K. (2019). Comparative Assessment of Vehicle Powertrain Concepts in the Early Development Phase, Berichte aus der Fahrzeugtechnik Shaker.
10. Koch, A., Bürchner, T., Herrmann, T., and Lienkamp, M. (2021). Eco-Driving for Different Electric Powertrain Topologies Considering Motor Efficiency. World Electr. Veh. J., 12.10.3390/wevj12010006 ↗
11. A real-time approach to robust identification of tyre–road friction characteristics on mixed- roads10.1080/00423114.2018.1504974 ↗
12. Nonlinear state and tire force estimation for advanced vehicle control10.1109/87.370717 ↗
13. Delay-Dependent Criteria for Robust Dynamic Stability Control of Articulated Vehicles
14. Measures of agreement between computation and experiment: Validation metrics10.1016/j.jcp.2006.03.037 ↗
15. A Bayesian approach for quantification of model uncertainty10.1016/j.ress.2010.02.015 ↗
16. A historical review of the development of verification and validation theories for simulation models10.1142/S1793962317300011 ↗
17. Validation in Simulation: Various Positions in the Philosophy of Science10.1287/mnsc.44.8.1087 ↗
18. Popper, K.R. (2006). Conjectures and Refutations: The Growth of Scientific Knowledge, Routledge. Routledge Classics.
19. Popper, K.R. (2008). The Logic of Scientific Discovery, Routledge. Routledge Classics.
20. Verification, validation, and confirmation of numerical models in the Earth sciences10.1126/science.263.5147.641 ↗
21. Oberkampf, W.L., and Roy, C.J. (2010). Verification and Validation in Scientific Computing, Cambridge University Press.10.1017/CBO9780511760396 ↗
22. IEEE (1990). IEEE Standard Glossary of Software Engineering Terminology, Institute of Electrical and Electronics Engineers Incorporated.
23. (2021, February 23). DIN EN ISO 9000:2015-11, Qualitätsmanagementsysteme—Grundlagen und Begriffe (ISO 9000:2015). Available online: https://link.springer.com/chapter/10.1007/978-3-658-27004-9_3.
24. ASME (2007). An overview of the PTC 60/V&V 10: Guide for verification and validation in computational solid mechanics. Eng. Comput., 23, 245–252.10.1007/s00366-007-0072-z ↗
25. DoD (1994). DoD Directive No. 5000.59: Modeling and Simulation (M&S) Management, Deparment of Defense.
26. Neelamkavil, F. (1994). Computer Simulation and Modelling, Wiley.
27. Sargent, R.G. (1979, January 3–5). Validation and Verification of Simulation Models. Proceedings of the 1979 Winter Simulation Conference, San Diego, CA, USA.
28. Verification and validation in computational fluid dynamics10.1016/S0376-0421(02)00005-2 ↗
29. Verification and validation of simulation models10.1057/jos.2012.20 ↗
30. Unified Framework and Survey for Model Verification, Validation and Uncertainty Quantification
31. Quantitative model validation techniques: New insights10.1016/j.ress.2012.11.011 ↗
32. ASME (2009). Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer: An American National Standard, The American Society of Mechanical Engineers.
33. Uncertainty quantification in vehicle dynamics
34. On the statistical mechanics of structural vibration10.1016/j.jsv.2019.115034 ↗
35. Separation of aleatory and epistemic uncertainty in probabilistic model validation10.1016/j.ress.2015.10.003 ↗
36. Model validation and predictive capability for the thermal challenge problem10.1016/j.cma.2007.07.030 ↗
37. Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems10.1016/j.ress.2015.01.023 ↗
38. Easterling, R.G. (2001). Measuring the Predictive Capability of Computational Methods: Principles and Methods, Issues and Illustrations, Sandia National Labs.10.2172/780290 ↗
39. Viehof, M. (2018). Objektive Qualitätsbewertung von Fahrdynamiksimulationen durch statistische Validierung. [Ph.D. Thesis, Technical University of Darmstadt].
40. A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing10.1016/j.cma.2011.03.016 ↗
41. Bayesian networks for system reliability reassessment10.1016/S0167-4730(01)00017-0 ↗
42. Riedmaier, S., Schneider, J., Danquah, B., Schick, B., and Diermeyer, F. (2021). Non-deterministic Model Validation Methodology for Simulation-based Safety Assessment of Automated Vehicles. Sim. Model. Prac. Theory, 94.10.1016/j.simpat.2021.102274 ↗
43. Calibration, validation, and sensitivity analysis: What’s what10.1016/j.ress.2005.11.031 ↗
44. Danquah, B. (2021, February 21). VVUQ Framework. Available online: https://github.com/TUMFTM/VVUQ-Framework.
45. Wacker, P., Adermann, J., Danquah, B., and Lienkamp, M. (2017, January 11–13). Efficiency determination of active battery switching technology on roller dynamometer. Proceedings of the Twelfth International Conference on Ecological Vehicles and Renewable Energies, Monte Carlo, Monaco.10.1109/EVER.2017.7935866 ↗
46. Danquah, B., Koch, A., Weis, T., Lienkamp, M., and Pinnel, A. (2019, January 8–10). Modular, Open Source Simulation Approach: Application to Design and Analyze Electric Vehicles. Proceedings of the 2019 Fourteenth International Conference on Ecological Vehicles and Renewable Energies (EVER), Monte-Carlo, Monaco.10.1109/EVER.2019.8813568 ↗
47. Danquah, B., Koch, A., Pinnel, A., Weiß, T., and Lienkamp, M. (2021, February 21). Component Library for Entire Vehicle Simulations. Available online: https://github.com/TUMFTM/Component_Library_for_Full_Vehicle_Simulations.
48. Minnerup, K., Herrmann, T., Steinstraeter, M., and Lienkamp, M. (2019). Case Study of Holistic Energy Management Using Genetic Algorithms in a Sliding Window Approach. World Electr. Veh. J., 10.10.3390/wevj10020046 ↗
49. Steinstraeter, M., Lewke, M., Buberger, J., Hentrich, T., and Lienkamp, M. (2020). Range Extension via Electrothermal Recuperation. World Electr. Veh. J., 11.10.3390/wevj11020041 ↗
50. Schmid, W., Wildfeuer, L., Kreibich, J., Buechl, R., Schuller, M., and Lienkamp, M. (2019, January 8–10). A Longitudinal Simulation Model for a Fuel Cell Hybrid Vehicle: Experimental Parameterization and Validation with a Production Car. Proceedings of the 2019 Fourteenth International Conference on Ecological Vehicles and Renewable Energies (EVER), Monte Carlo, Monaco.10.1109/EVER.2019.8813648 ↗
51. Kalt, S., Erhard, J., Danquah, B., and Lienkamp, M. (2019, January 8–10). Electric Machine Design Tool for Permanent Magnet Synchronous Machines. Proceedings of the 2019 Fourteenth International Conference on Ecological Vehicles and Renewable Energies (EVER), Monte Carlo, Monaco.10.1109/EVER.2019.8813601 ↗
52. Completed Richardson extrapolation in space and time10.1002/(SICI)1099-0887(199707)13:7<573::AID-CNM84>3.0.CO;2-6 ↗
53. Wacker, P., Wheldon, L., Sperlich, M., Adermann, J., and Lienkamp, M. (2017, January 22–24). Influence of active battery switching on the drivetrain efficiency of electric vehicles. Proceedings of the 2017 IEEE Transportation Electrification Conference (ITEC), Chicago, IL, USA.10.1109/ITEC.2017.7993243 ↗
54. Devore, J.L. (2012). Probability and Statistics for Engineering and the Sciences, Brooks/Cole, Cengage Learning. [8th ed.].
55. Sample Size Required for Estimating the Standard Deviation as a Per Cent of its True Value10.1080/01621459.1950.10483356 ↗
56. A Holistic Approach to Uncertainty Quantification with Application to Supersonic Nozzle Thrust10.1615/Int.J.UncertaintyQuantification.2012003562 ↗
57. Schmeiler, S. (2020). Enhanced Insights from Vehicle Simulation by Analysis of Parametric Uncertainties, Fahrzeugtechnik, Dr. Hut.
58. Wilson, A., and Adams, R.A. (2013, January 17–19). Gaussian Process Kernels for Pattern Discovery and Extrapolation. Proceedings of the 30th International Conference on Machine Learning, PMLR, Atlanta, GA, USA.
59. Construction of bootstrap confidence intervals on sensitivity indices computed by polynomial chaos expansion10.1016/j.ress.2013.09.011 ↗
60. Applications of the Theory of Error to the Interpretation of Trends
61. Miller, R.G. (1981). Simultaneous Statistical Inference, Springer.10.1007/978-1-4613-8122-8 ↗
62. Prediction Regions for Several Predictions from a Single Regression Line10.1080/00401706.1961.10489924 ↗
63. Gaussian process for nonstationary time series prediction10.1016/j.csda.2004.02.006 ↗