Creating Collaborative Augmented Reality Experiences for Industry 4.0 Training and Assistance Applications: Performance Evaluation in the Shipyard of the Future
Source abstract
Abstract
Industrial Augmented Reality (IAR) is one of the key technologies pointed out by the Industry 4.0 paradigm as a tool for improving industrial processes and for maximizing worker efficiency. Training and assistance are two of the most popular IAR-enabled applications, since they may significantly facilitate, support, and optimize production and assembly tasks in industrial environments. This article presents an IAR collaborative application developed jointly by Navantia, one of the biggest European shipbuilders, and the University of A Coruña (Spain). The analysis, design, and implementation of such an IAR application are described thoroughly so as to enable future developers to create similar IAR applications. The IAR application is based on the Microsoft HoloLens smart glasses and is able to assist and to guide shipyard operators during their training and in assembly tasks. The proposed IAR application embeds a novel collaborative protocol that allows operators to visualize and interact in a synchronized way with the same virtual content. Thus, all operators that share an IAR experience see each virtual object positioned at the same physical spot and in the same state. The collaborative application is first evaluated and optimized in terms of packet communications delay and anchor transmission latency, and then, its validation in a shipyard workshop by Navantia’s operators is presented. The performance results show fast response times for regular packets (less than 5 ms), low interference rates in the 5 GHz band, and an anchor transmission latency of up to 30 s. Regarding the validation tests, they allow for obtaining useful insights and feedback from the industrial operators, as well as clear guidelines that will help future developers to face the challenges that will arise when creating the next generation of IAR applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Augmented Reality Applications · IoT and Edge/Fog Computing · Advanced Manufacturing and Logistics Optimization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Faculty of Computer Science, Universidade da Coruña, 15071 A Coruña, Spain
Centro de Investigación CITIC, Universidade da Coruña, 15071 A Coruña, Spain
Navantia S. A., Astillero de Ferrol, 15403 Ferrol, Spain
Funding
Centro de Investigación de Galicia "CITIC", funded by Xunta de Galicia and the European Union (European Regional Development Fund- Galicia 2014-2020 Program)
References (63)
Show source-provided references
1. (2020, December 05). Announcement of the Industrie 4.0 Project in the 2011 Hannover Fair. Available online: https://www.vdi-nachrichten.com/Technik-Gesellschaft/Industrie-40-Mit-Internet-Dinge-Weg-4-industriellen-Revolution.
2. (2020, December 05). Industrie 4.0. Available online: https://www.bmbf.de/de/zukunftsprojekt-industrie-4-0-848.html.
3. Digital Twin and Big Data Towards Smart Manufacturing and Industry 4.0: 360 Degree Comparison10.1109/ACCESS.2018.2793265 ↗
4. Digital Twin: Values, Challenges and Enablers From a Modeling Perspective10.1109/ACCESS.2020.2970143 ↗
5. Wu, Y., Dai, H.-N., and Wang, H. (2020). Convergence of Blockchain and Edge Computing for Secure and Scalable IIoT Critical Infrastructures in Industry 4.0. IEEE Internet Things J.10.1109/JIOT.2020.3025916 ↗
6. A Review on the Application of Blockchain to the Next Generation of Cybersecure Industry 4.0 Smart Factories10.1109/ACCESS.2019.2908780 ↗
7. Industrial Artificial Intelligence for industry 4.0-based manufacturing systems10.1016/j.mfglet.2018.09.002 ↗
8. Challenges to IoT-Enabled Predictive Maintenance for Industry 4.010.1109/JIOT.2019.2957029 ↗
9. Context-Aware Cloud Robotics for Material Handling in Cognitive Industrial Internet of Things10.1109/JIOT.2017.2728722 ↗
10. Towards augmented reality manuals for industry 4.0: A methodology10.1016/j.rcim.2018.10.001 ↗
11. A review on industrial augmented reality systems for the industry 4.0 shipyard10.1109/ACCESS.2018.2808326 ↗
12. Sutherland, I.E. (1965, January 24–29). The ultimate display. Proceedings of the IFIP, New York, NY, USA.
13. Wireless Channel Assessment of Auditoriums for the Deployment of Augmented Reality Systems for Enhanced Show Experience of Impaired Persons
14. Effects of Gamified Augmented Reality in Public Spaces10.1109/ACCESS.2019.2945819 ↗
15. Kim, M., Park, K.B., Choi, S.H., Lee, J.Y., and Kim, D.Y. (2018, January 26–30). AR/VR-Based Live Manual for User-Centric Smart Factory Services. Proceedings of the IFIP International Conference on Advances in Production Management Systems, Seoul, Korea.10.1007/978-3-319-99707-0_52 ↗
16. Daling, L., Abdelrazeq, A., Sauerborn, C., and Hees, F. (2019, January 24–28). A Comparative Study of Augmented Reality Assistant Tools in Assembly. Proceedings of the International Conference on Applied Human Factors and Ergonomics, Washington, DC, USA.10.1007/978-3-030-19135-1_74 ↗
17. Smparounis, K., Mavrikios, D., Pappas, M., Xanthakis, V., Viganò, G.P., and Pentenrieder, K. (2008, January 23–28). A virtual and augmented reality approach to collaborative product design and demonstration. Proceedings of the 2008 IEEE International Technology Management Conference (ICE), Lisbon, Portugal.
18. Exploring the benefits of augmented reality documentation for maintenance and repair10.1109/TVCG.2010.245 ↗
19. Werrlich, S., Daniel, A., Ginger, A., Nguyen, P.A., and Notni, G. (2018, January 16–20). Comparing HMD-based and Paper-based Training. Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany.10.1109/ISMAR.2018.00046 ↗
20. Vidal-Balea, A., Blanco-Novoa, O., Fraga-Lamas, P., Vilar-Montesinos, M., and Fernández-Caramés, T.M. (2020). A Collaborative Augmented Reality Application for Training and Assistance during Shipbuilding Assembly Processes. Multidiscip. Digit. Publ. Inst. Proc., 54.10.3390/proceedings2020054004 ↗
21. (2020, November 04). Microsoft HoloLens Official Web Page. Available online: https://www.microsoft.com/en-us/hololens.
22. Augmented reality system for virtual training of parts assembly10.1016/j.proeng.2015.01.422 ↗
23. Tang, A., Owen, C., Biocca, F., and Mou, W. (2003, January 5). Comparative effectiveness of augmented reality in object assembly. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Fort Lauderdale, FL, USA.10.1145/642611.642626 ↗
24. Alesky, M., Vartiainen, E., Domova, V., and Naedele, M. (2014, January 13–16). Augmented reality for improved service delivery. Proceedings of the IEEE 28th International Conference on Advanced Information Networking and Applications, Victoria, BC, Canada.10.1109/AINA.2014.146 ↗
25. Using augmented reality to improve usability of the user interface for driving a telepresence robot
26. Herrera, K.A., Rocha, J.A., Silva, F.M., and Andaluz, V.H. (2020, January 24–27). Training Systems for Control of Mobile Manipulator Robots in Augmented Reality. Proceedings of the 2020 15th Iberian Conference on Information Systems and Technologies (CISTI), Sevilla, Spain.10.23919/CISTI49556.2020.9141012 ↗
27. Lapointe, J.F., Molyneaux, H., and Allili, M. (2020). A Literature Review of AR-Based Remote Guidance Tasks with User Studies, Springer.10.1007/978-3-030-49698-2_8 ↗
28. Virtual and augmented reality applications in manufacturing10.3182/20130619-3-RU-3018.00637 ↗
29. Chicaiza, E.A., De la Cruz, E.I., and Andaluz, V.H. (2018). Augmented Reality System for Training and Assistance in the Management of Industrial Equipment and Instruments. Lect. Notes Comput. Sci., 675–686.10.1007/978-3-030-03801-4_59 ↗
30. Reddy, K.P.K., Venkitesh, B., Varghese, A., Narendra, N., Chandra, G., and Balamuralidhar, P. (2015, January 23–25). Deformable 3D CAD models in mobile augmented reality for tele-assistance. Proceedings of the 2015 Asia Pacific Conference on Multimedia and Broadcasting, Kuta, Indonesia.10.1109/APMediaCast.2015.7210274 ↗
31. Schneider, M., Rambach, J., and Stricker, D. (2017, January 22–25). Augmented reality based on edge computing using the example of remote live support. Proceedings of the 2017 IEEE International Conference on Industrial Technology (ICIT), Toronto, ON, Canada.10.1109/ICIT.2017.7915547 ↗
32. Augmented reality for construction site monitoring and documentation10.1109/JPROC.2013.2294314 ↗
33. Moloney, J. (2006, January 5–7). Augmented reality visualisation of the built environment to support design decision making. Proceedings of the Tenth International Conference on Information Visualisation (IV’06), London, UK.
34. Olfactory-Based Augmented Reality Support for Industrial Maintenance10.1109/ACCESS.2020.2970220 ↗
35. Lampen, E., Lehwald, J., and Pfeiffer, T. (2020). A Context-Aware Assistance Framework for Implicit Interaction with an Augmented Human. Virtual, Augmented and Mixed Reality. Industrial and Everyday Life Applications, Proceedings of the 12th International Conference, VAMR 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, 19–24 July 2020, Springer. Part II.10.1007/978-3-030-49698-2_7 ↗
36. Blanco-Novoa, Ó., Fraga-Lamas, P., Vilar-Montesinos, M.A., and Fernández-Caramés, T.M. (2020). Creating the Internet of Augmented Things: An Open-Source Framework to Make IoT Devices and Augmented and Mixed Reality Systems Talk to Each Other. Sensors, 20.10.3390/s20113328 ↗
37. Digital Twin in the IoT Context: A Survey on Technical Features, Scenarios, and Architectural Models10.1109/JPROC.2020.2998530 ↗
38. Effects of augmented reality on the performance of teleoperated industrial assembly tasks in a robotic embodiment10.1109/THMS.2017.2782490 ↗
39. Presence as being-in-the-world10.1162/105474698565541 ↗
40. Rosenberg, L.B. (1993, January 18–22). Virtual fixtures: Perceptual tools for telerobotic manipulation. Proceedings of the IEEE Virtual Reality Annual International Symposium, Seattle, WA, USA.10.1109/VRAIS.1993.380795 ↗
41. Gwilliam, J.C., Mahvash, M., Vagvolgyi, B., Vacharat, A., Yuh, D.D., and Okamura, A.M. (2009, January 12–17). Effects of haptic and graphical force feedback on teleoperated palpation. Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan.10.1109/ROBOT.2009.5152705 ↗
42. Livingston, M.A., Gabbard, J.L., Swan, J.E., Sibley, C.M., and Barrow, J.H. Basic perception in head-worn augmented reality displays. Proceedings of the Human Factors in Augmented Reality Environments.
43. The perception of egocentric distances in virtual environments-a review10.1145/2543581.2543590 ↗
44. Smith, E., Semple, G., Evans, D., McRae, K., and Blackwell, P. (2020, January 19–24). Augmented Instructions: Analysis of Performance and Efficiency of Assembly Tasks. Proceedings of the 12th International Conference, VAMR 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark. Lecture Notes in Computer Science.
45. Collaborative welding system using BIM for robotic reprogramming and spatial augmented reality10.1016/j.autcon.2019.04.020 ↗
46. Lee, J., Kwon, O., Choi, J., and Park, C. (2012, January 23–25). A Study on Construction Defect Management Using Augmented Reality Technology. Proceedings of the 2012 International Conference on Information Science and Applications, Suwon, Korea.10.1109/ICISA.2012.6220933 ↗
47. Reduction of piping management person-hours through use of AR technology at shipbuilding sites
48. Olbrich, M., Wuest, H., Riess, P., and Bockholt, U. (2011, January 26–29). Augmented reality pipe layout planning in the shipbuilding industry. Proceedings of the 2011 10th IEEE International Symposium Mixed and Augmented Reality (ISMAR), Basel, Switzerland.10.1109/ISMAR.2011.6162911 ↗
49. Ding, J., Zhu, Y., Luo, M., Zhu, M., Fan, X., and Zhou, Z. (2020). AR Assisted Process Guidance System for Ship Block Fabrication. Virtual, Augmented and Mixed Reality. Industrial and Everyday Life Applications, Springer International Publishing.10.1007/978-3-030-49698-2_5 ↗
50. A practical evaluation of commercial industrial augmented reality systems in an industry 4.0 shipyard10.1109/ACCESS.2018.2802699 ↗
51. (2020, November 05). Atos Air Platform Official Web Page. Available online: https://atos.net/nl/nederland/augmented-interactive-reality.
52. (2020, November 05). Andy 3D Official Web Page. Available online: https://www.capgemini.com/gb-en/resources/andy3d-remote-assistance-and-asset-revamping-web-platform/.
53. (2020, November 05). Reflekt Remote Official Web Page. Available online: https://www.re-flekt.com/.
54. (2020, November 05). Librestream Official Web Page. Available online: https://librestream.com/.
55. (2020, November 05). Remote Eye Official Web Page. Available online: https://remoteeye.com/.
56. (2020, November 05). ATR by Innovae Official Web Page. Available online: https://www.innovae.eu/atr-technical-remote-assistant/?lang=en.
57. Sereno, M., Wang, X., Besancon, L., Mcguffin, M.J., and Isenberg, T. (2020). Collaborative Work in Augmented Reality: A Survey. IEEE Trans. Vis. Comput. Graph.10.1109/TVCG.2020.3032761 ↗
58. (2020, December 02). Unity Official Web Page. Available online: https://unity.com/.
59. (2020, December 02). Siemens NX Official Web Page. Available online: https://www.plm.automation.siemens.com/global/en/products/nx/.
60. (2020, December 02). Blender Official Web Page. Available online: https://www.blender.org/.
61. (2020, December 02). Microsoft Recommendations for Model Optimization. Available online: https://docs.microsoft.com/en-us/dynamics365/mixed-reality/import-tool/best-practices.
62. (2020, December 02). Universal Windows Platform Official Web Page. Available online: https://visualstudio.microsoft.com/es/vs/features/universal-windows-platform/.
63. ITU (2014). The Tactile Internet. ITU-T Technology Report, ITU. Available online: https://www.itu.int/dms_pub/itu-t/opb/gen/T-GEN-TWATCH-2014-1-PDF-E.pdf.