Influence of Virtual Reality on User Evaluation of Prototypes in the Development Process—A Comparative Study with Control Rooms for Onshore Drilling Rigs
Source abstract
Abstract
User evaluations of prototypes in virtual reality (VR) offer high potential for products that require resource-intensive prototype construction, such as drilling rigs. This study examined whether the user evaluation of a VR prototype for controlling an onshore drilling rigproduces results comparable to an evaluation in the real world. Using a between-subject design, 16 drilling experts tested a prototype in VR and reality. The experts performed three different work processes and evaluated their satisfaction based on task performance, user experience, and usability via standardized questionnaires. A test leader evaluated the effectiveness of the work process execution using a 3-level rating scheme. The number of user interactions and time on task were recorded. There were no significant differences in the effectiveness, number of interactions required, perceived usability, and satisfaction with respect to task performance. In VR, the drilling experts took significantly more time to complete tasks and rated the efficiency of the VR prototype significantly higher. Overall, the real-world evaluation provided more insights into prototype optimization. Nevertheless, several usability issues have been identified in VR. Therefore, user evaluations in VR are particularly suitable in the early development phases to identify usability issues, without the need to produce real prototypes.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Virtual Reality Applications and Impacts · Human-Automation Interaction and Safety · Augmented Reality Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Center for Ergonomics and Medical Engineering, FH Münster University of Applied Sciences, Bürgerkamp 3, 48565 Steinfurt, Germany
Institute of Psychology and Ergonomics, Technical University Berlin, Fasanenstraße 1, 10623 Berlin, Germany
Funding
Bentec GmbH Drilling & Oilfield Systems
References (58)
Show source-provided references
1. (2019). Ergonomics of Human-System Interaction—Part 220: Processes for Enabling, Executing and Assessing Human-Centred Design within Organizations (Standard No. ISO 9241-220:2019).
2. Determining fidelity of mixed prototypes: Effect of media and physical interaction10.1016/j.apergo.2019.05.007 ↗
3. Virtual prototyping and testing of in-vehicle interfaces10.1080/00140130303528 ↗
4. Fischer, H., and Hess, S. (2019). Mensch und Computer 2019—Usability Professionals, Gesellschaft für Informatik e.V. Und German UPA e.V.
5. De Freitas, F.V., Gomes, M.V.M., and Winkler, I. (2022). Benefits and Challenges of Virtual-Reality-Based Industrial Usability Testing and Design Reviews: A Patents Landscape and Literature Review. Appl. Sci., 12.10.3390/app12031755 ↗
6. Optimizing the Design Review Process for Cyber-Physical Systems using Virtual Reality10.1016/j.procir.2020.03.115 ↗
7. Duffy, V.G. (2019). Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body and Motion, Springer International Publishing.10.1007/978-3-030-22216-1 ↗
8. Turunen, M., Väätäjä, H., Paavilainen, J., and Olsson, T. (2017, January 20–21). Virtual Prototyping in Design Reviews of Industrial Systems. Proceedings of the 21st International Annual Academic Mindtrek Conference (AcademicMindtrek’17), Tampere, Finland.
9. Suitability of virtual prototypes to support human factors/ergonomics evaluation during the design10.1016/j.apergo.2016.02.015 ↗
10. An Industry Case Study: Investigating Early Design Decision Making in Virtual Reality10.1115/1.4034267 ↗
11. Implications of Virtual Reality on Environmental Sustainability in Manufacturing Industry: A Case Study10.1016/j.procir.2021.11.078 ↗
12. User Centered Design of Interaction Techniques for VR-Based Automotive Design Reviews10.3389/frobt.2019.00013 ↗
13. Analyzing the potential of Virtual Reality for engineering design review10.1016/j.autcon.2019.03.018 ↗
14. Backhaus, C. (2010). Usability-Engineering in der Medizintechnik, Springer.10.1007/978-3-642-00511-4 ↗
15. Spencer, S.N., Morishima, S., Itoh, Y., Shiratori, T., Yue, Y., and Lindeman, R. (December, January 28). Comparison of the usability of a car infotainment system in a mixed reality environment and in a real car. Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (VRST ’18), Tokyo, Japan.
16. A Transdisciplinary digital approach for tractor’s human-centred design10.1080/0951192X.2019.1599441 ↗
17. Krömker, H. (2019). HCI in Mobility, Transport, and Automotive Systems, Springer International Publishing.10.1007/978-3-030-22666-4 ↗
18. Harrison, S., Bardzell, S., Neustaedter, C., and Tatar, D. (2019, January 23–28). Virtually the Same Experience?. Proceedings of the 2019 On Designing Interactive Systems Conference (DIS ’19), San Diego, CA, USA.
19. Bruno, F., Cosco, F., Angilica, A., and Muzzupappa, M. (2011). Volume 3: 30th Computers and Information in Engineering Conference, Parts A and B, Proceedings of the ASME 2010 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (ASMEDC), Montreal, QC, Canada, 15–18 August 2010, ASME.
20. Krömker, H. (2020). HCI in Mobility, Transport, and Automotive Systems. Automated Driving and In-Vehicle Experience Design, Springer International Publishing.10.1007/978-3-030-50523-3 ↗
21. Virtual prototyping in the design of see-through features in mobile machinery10.1007/s10055-019-00384-y ↗
22. Use of Immersive 3-D Virtual Reality Environments in Control Room Validations10.1080/00295450.2017.1420335 ↗
23. Drillers’ Cognitive Skills Monitoring Task
24. Virtual Reality Applied to Product Development in the Oil and Gas Industry: A Brief Review10.34178/jbth.v5i4.259 ↗
25. Does virtual reality simulation have a role in training trauma and orthopaedic surgeons?10.1302/0301-620X.100B5.BJJ-2017-1439 ↗
26. Validation of video-based skill assessment in carotid artery stenting10.1016/j.ejvs.2009.03.008 ↗
27. van der Veer, G., and Gale, C. (2005). CHI ‘05 Extended Abstracts on Human Factors in Computing Systems, Proceedings of the CHI05: CHI 2005 Conference on Human Factors in Computing Systems, Portland, OR, USA, 2–7 April 2005, ACM.
28. Kiss, M., Schmidt, G., and Babbel, E. (2008, January 7–8). Das Wizard of Oz Fahrzeug: Rapid prototyping und usability Testing von zukünftigen Fahrerassistenzsystemen. Proceedings of the 3 Tagung Aktive Sicherheit durch Fahrerassistenz, München, Germany.
29. Janssen, C.P., Donker, S.F., Chuang, L.L., and Ju, W. (2019, January 21–25). Using the wizard of Oz paradigm to prototype automated vehicles. Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings (AutomotiveUI ’19), Utrecht, The Netherlands.
30. Buja, H.-O. (2011). Handbuch der Tief-, Flach-, Geothermie- und Horizontalbohrtechnik: Bohrtechnik in Grundlagen und Anwendung; mit 119 Tabellen, Vieweg + Teubner. [1st ed.].
31. Reich, M. (2022). Auf Jagd im Untergrund: Mit Hightech auf der Suche nach Öl, Gas und Erdwärme, Springer. [3rd ed.].10.1007/978-3-662-64151-4 ↗
32. Nielsen, J. (2023, July 12). Success Rate: The Simplest Usability Metric. Available online: https://www.nngroup.com/articles/success-rate-the-simplest-usability-metric/.
33. Holzinger, A. (2008). HCI and Usability for Education and Work, Springer.10.1007/978-3-540-89350-9 ↗
34. Brau, H., Lehmann, A., Petrovic, K., and Schroeder, M.C. (2013). Tagungsband UP13, German UPA e.V.
35. SUS: A quick and dirty usability scale
36. Determining what individual SUS scores mean: Adding an adjective rating scale
37. Mey, G., and Mruck, K. (2010). Handbuch Qualitative Forschung in der Psychologie, VS Verlag für Sozialwissenschaften.10.1007/978-3-531-92052-8 ↗
38. Toward a Theory of Situation Awareness in Dynamic Systems10.1518/001872095779049543 ↗
39. Measuring Situation Awareness in complex systems: Comparison of measures study10.1016/j.ergon.2008.10.010 ↗
40. How to recognise a kick: A cognitive task analysis of drillers’ situation awareness during well operations10.1016/j.jlp.2016.07.003 ↗
41. Operator situation awareness and physiological states during offshore well control scenarios10.1016/j.jlp.2018.07.010 ↗
42. Comes, T., Fiedrich, S., Fortier, S., and Geldermann, J. (2013, January 12–15). Comparing performance and situation awareness in USAR unit tasks in a virtual and real environment. Proceedings of the 10th International Conference on Information Systems for Crisis Response and Management 2013 (ISCRAM 2013), Baden Baden, Germany.
43. Task Performance and Situation Awareness with a Virtual Reality Head-Mounted Display10.1177/1541931213602008 ↗
44. Salas, E., and Dietz, A.S. (2017). Situational Awareness, Routledge.10.4324/9781315087924 ↗
45. Clark, R.E., and Sugrue, B.M. (1988). Educational Media and Technology Yearbook, Libraries Unlimited, Inc.
46. Olsen, D.R., Arthur, R.B., Hinckley, K., Morris, M.R., Hudson, S., and Greenberg, S. (2009, January 4–9). User Experience over Time. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09), Boston, MA, USA.
47. Duarte, E., and Rosa, C. (2022). Developments in Design Research and Practice, Springer International Publishing.10.1007/978-3-030-86596-2 ↗
48. Effects of virtual reality and test environment on user experience, usability, and mental workload in the evaluation of a blood pressure monitor10.3389/frvir.2023.1151190 ↗
49. The influence of design aesthetics in usability testing: Effects on user performance and perceived usability10.1016/j.apergo.2009.09.002 ↗
50. Chandran, S.K., Forbes, J., Bittick, C., Allanson, K., and Brinda, F. (2017). Sample Size in the Application of System Usability Scale to Automotive Interfaces, SAE.10.4271/2017-01-1383 ↗
51. An Empirical Evaluation of the System Usability Scale10.1080/10447310802205776 ↗
52. Tullis, T., and Stetson, J. (2004). A Comparison of Questionnaires for Assessing Website Usability, Usability Professionals Association (UPA).
53. Simoes, F., Bezerra, M., Teixeira, J.M., Correia, W., and Teichrieb, V. (2016, January 21–24). A User Perspective Analysis on Augmented vs. 3D Printed Prototypes for Product’s Project Design. Proceedings of the 2016 XVIII Symposium on Virtual and Augmented Reality (SVR), Gramado, Brazil.10.1109/SVR.2016.22 ↗
54. Burduk, A., Chlebus, E., Nowakowski, T., and Tubis, A. (2019). Intelligent Systems in Production Engineering and Maintenance, Springer International Publishing.10.1007/978-3-319-97490-3 ↗
55. Design and Analysis of New Haptic Joysticks for Enhancing Operational Skills in Excavator Control10.1115/1.4048462 ↗
56. Anguelov, N. (2009). Haptic and Acoustic Parameters for Objectifying and Optimizing the Perceived Value of Switches and Control Panels for Automotive Interiors. [Ph.D. Thesis, Technical University Dresden].
57. Perception of products by progressive multisensory integration. A study on hammers10.1016/j.apergo.2010.09.014 ↗
58. Using virtual reality to assess user experience10.1177/0018720812465006 ↗