PERFORM: A Metric for Evaluating Autonomous System Performance in Marine Testbed Environments Using Interval Type-2 Fuzzy Logic
Source abstract
Abstract
Trust and confidence in autonomous behavior is required to send autonomous vehicles into operational missions. The authors introduce the Performance Evaluation and Review Framework Of Robotic Missions (PERFORM), a framework to enable a rigorous and replicable autonomy test environment, thereby filling the void between that of merely simulating autonomy and that of completing true field missions. A generic architecture for defining the missions under test is proposed and a unique Interval Type-2 Fuzzy Logic approach is used as the foundation for the mathematically rigorous autonomy evaluation framework. The test environment is designed to aid in (1) new technology development (i.e., providing direct comparisons and quantitative evaluations between autonomy algorithms), (2) the validation of the performance of specific autonomous platforms, and (3) the selection of the appropriate robotic platform(s) for a given mission type (e.g., for surveying, surveillance, search and rescue). Three case studies are presented to apply the metric to various test scenarios. Results demonstrate the flexibility of the technique with the ability to tailor tests to the user’s design requirements accounting for different priorities related to acceptable risks and goals of a given mission.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Maritime Navigation and Safety · Robotic Path Planning Algorithms · Underwater Vehicles and Communication Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Marine Science and Engineering, University of New Hampshire, Durham, NH 03824, USA
Woods Hole Oceanographic Institution, Woods Hole, MA 02543, USA
Naval Undersea Warfare Center—Division Keyport, Keyport, WA 98345, USA
Funding
Naval Sea Systems Command · Naval Engineering Education Consortium
References (40)
Show source-provided references
1. Bouchard, A., and Tatum, R. (2015, January 19–22). Verification of Autonomous Systems: Challenges of the Present and Areas for Exploration. Proceedings of the OCEANS 2015—MTS/IEEE, Washington, DC, USA.10.23919/OCEANS.2015.7404475 ↗
2. Dutta, R., Guo, X., and Jin, Y. (2016, January 6–9). Quantifying Trust in Autonomous System Under Uncertainties. Proceedings of the 29th IEEE International System-on-Chip Conference (SOCC), Seattle, WA, USA.10.1109/SOCC.2016.7905511 ↗
3. Unmanned surface vehicles: An overview of developments and challenges10.1016/j.arcontrol.2016.04.018 ↗
4. A multi-layered fast marching method for unmanned surface vehicle path planning in a time-variant maritime environment10.1016/j.oceaneng.2016.11.009 ↗
5. Grotli, E., Reinen, T., Grythe, K., Transeth, A., Vagia, M., Bjerkeng, M., Rundtop, P., Svendsen, E., Rodseth, O., and Eidnes, G. (2015, January 19–22). Seatonomy. Proceedings of the OCEANS 2015—MTS/IEEE, Washington, DC, USA.10.23919/OCEANS.2015.7401827 ↗
6. Delivering test and evaluation tools for autonomous unmanned vehicles to the fleet
7. Redfield, S., and Seto, M. (2017). Verification challenges for autonomous systems. Autonomy and Artificial Intelligence: A Threat or Savior? Lawless, W., Mittu, R., Sofge, D., Russell, S. Eds., Springer International Publishing.10.1007/978-3-319-59719-5_5 ↗
8. Betts, K., and Petty, M. (2016). Automated search-based robustness testing for autonomous vehicle software. Model. Simul. Eng., 2016.10.1155/2016/5309348 ↗
9. Mullins, G., Stankiewicz, P., and Gupta, S. (June, January 29). Automated generation of diverse and challenging scenarios for test and evaluation of autonomous vehicles. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989173 ↗
10. Durst, P., and Gray, W. (2020, May 13). Levels of Autonomy and Autonomous System Performance Assessment for Intelligent Unmanned Systems. Available online: https://apps.dtic.mil/dtic/tr/fulltext/u2/a601656.pdf.
11. Müller, J., Ketter, W., Kaminka, G., Wagner, G., and Bulling, N. (2015). A Metrics Framework for Quantifying Autonomy in Complex Systems. Multiagent System Technologies, Springer International Publishing.10.1007/978-3-319-27343-3 ↗
12. Olszewska, J., Barreto, M., Bermejo-Alonso, J., Carbonera, J., Chibani, A., Fiorini, S., Goncalves, P., Habib, M., Khamis, A., and Olivares, A. (September, January 28). Ontology for Autonomous Robotics. Proceedings of the 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Lisbon, Portugal.
13. (2020, May 13). Safety Report. Available online: https://waymo.com/safety/.
14. Menzel, T., Bagschik, G., and Maurer, M. (2018, January 26–30). Scenarios for Development, Test and Validation of Automated Vehicles. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.10.1109/IVS.2018.8500406 ↗
15. Kalra, N., and Paddock, S. (2016). Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?, RAND Corporation. Available online: https://www.rand.org/pubs/research_reports/RR1478.html.10.7249/RR1478 ↗
16. Huang, H. (2007, January 28–30). Autonomy levels for unmanned systems (ALFUS) framework: Safety and application issues. Proceedings of the 2007 Workshop on Performance Metrics for Intelligent Systems, Baltimore, MA, USA.10.1145/1660877.1660883 ↗
17. Durst, P., Gray, W., Nikitenko, A., Caetano, J., Trentini, M., and King, R. (2014, January 14–18). A framework for predicting the mission-specific performance of autonomous unmanned systems. Proceedings of the 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, USA.10.1109/IROS.2014.6942823 ↗
18. Test and Evaluation of Autonomous Ground Vehicles10.1155/2014/681326 ↗
19. Xiong, G., Zhao, X., Liu, H., Wu, S., Gong, J., Zhang, H., Tan, H., and Chen, H. (2010, January 21–24). Research on the Quantitative Evaluation System for Unmanned Ground Vehicles. Proceedings of the 2010 IEEE Intelligent Vehicles Symposium, San Diego, CA, USA.10.1109/IVS.2010.5548144 ↗
20. Type-2 Fuzzy Sets: Some Questions and Answers
21. Linda, O., and Manic, M. (2011, January 11–15). Uncertainty Modeling for Interval Type-2 Fuzzy Logic Systems Based on Sensor Characteristics. Proceedings of the 2011 IEEE Symposium on Advances in Type-2 Fuzzy Logic Systems (T2FUZZ), Paris, France.10.1109/T2FUZZ.2011.5949550 ↗
22. Uncertainty Measures for Interval Type-2 Fuzzy Sets10.1016/j.ins.2007.07.012 ↗
23. Garibaldi, J., Musikasuwan, S., and Ozen, T. (2005, January 25–27). The Association between Non-Stationary and Interval Type-2 Fuzzy Sets: A Case Study. Proceedings of the 14th IEEE International Conference on Fuzzy Systems, Reno, NV, USA.
24. The concept of a linguistic variable and its application to approximate reasoning10.1016/0020-0255(75)90036-5 ↗
25. Type-2 Fuzzy Logic Systems10.1109/91.811231 ↗
26. Roberts, W., Meyer, P., Seifert, S., Evans, E., Steffens, M., and Mavris, D. (2018, January 22–25). A Flexible Problem Definition for Event-Based Missions for Evaluation of Autonomous System Behavior. Proceedings of the OCEANS 2018 MTS/IEEE, Charleston, SC, USA.10.1109/OCEANS.2018.8604771 ↗
27. Zimmermann, H. (2001). Fuzzy Set Theory and Its Applications, Springer. [4th ed.].10.1007/978-94-010-0646-0 ↗
28. Fuzzy Sets10.1016/S0019-9958(65)90241-X ↗
29. A comparative study of type-1 fuzzy logic systems, interval type-2 fuzzy logic systems and generalized type-2 fuzzy logic systems in control problems10.1016/j.ins.2016.03.026 ↗
30. Castillo, O., and Aguilar, L. (2019). Type-2 Fuzzy Logic in Control of Nonsmooth Systems: Theoretical Concepts and Applications, Springer International Publishing.10.1007/978-3-030-03134-3 ↗
31. General Type-2 Fuzzy Logic Systems Made Simple: A Tutorial10.1109/TFUZZ.2013.2286414 ↗
32. Type-2 Fuzzy Sets and Systems: An Overview
33. Interval Type-2 Fuzzy Sets are Generalization of Interval-Valued Fuzzy Sets: Toward a Wider View on Their Relationship10.1109/TFUZZ.2014.2362149 ↗
34. Centroid of a type-2 fuzzy set10.1016/S0020-0255(01)00069-X ↗
35. Mendel, J., Hagras, H., Tan, W., Melek, W., and Ying, H. (2014). Introduction to Type-2 Fuzzy Logic Control, IEEE Press.10.1002/9781118886540 ↗
36. Dalpe, A., Cook, A., Thein, M.-W., and Renken, M. (2018, January 22–25). A Multi-Layered Approach to Autonomous Surface Vehicle Map-Based Autonomy. Proceedings of the OCEANS 2018 MTS/IEEE, Charleston, SC, USA.10.1109/OCEANS.2018.8604602 ↗
37. Probabilistic roadmaps for path planning in high-dimensional configuration spaces10.1109/70.508439 ↗
38. Benatar, N., Aickelin, U., and Garibaldi, J. (2013). Performance Measurement Under Increasing Environmental Uncertainty in the Context of Interval Type-2 Fuzzy Logic Based Robotic Sailing. arXiv.10.2139/ssrn.2828470 ↗
39. (2020, May 12). Available online: https://marvelmind.com.
40. (2020, June 11). Available online: https://nauticalcharts.noaa.gov/publications/standards-and-requirements.html.