Extended Reality Application Framework for a Digital-Twin-Based Smart Crane
Source abstract
Abstract
Industry 4.0 is moving forward under technology upgrades, utilizing information technology to improve the intelligence of the industry, whereas Industry 5.0 is value-driven, aiming to focus on essential societal needs, values, and responsibility. The manufacturing industry is currently moving towards the integration of productivity enhancements and sustainable human employment. Such a transformation has deeply changed the human–machine interaction (HMI), among which digital twin (DT) and extended reality (XR) are two cutting-edge technologies. A manufacturing DT offers an opportunity to simulate, monitor, and optimize the machine. In the meantime, XR empowers HMI in the industrial field. This paper presents an XR application framework for DT-based services within a manufacturing context. This work aims to develop a technological framework to improve the efficiency of the XR application development and the usability of the XR-based HMI systems. We first introduce four layers of the framework, including the perception layer with the physical machine and its ROS-based simulation model, the machine communication layer, the network layer containing three kinds of communication middleware, and the Unity-based service layer creating XR-based digital applications. Subsequently, we conduct the responsiveness test for the framework and describe several XR industrial applications for a DT-based smart crane. Finally, we highlight the research challenges and potential issues that should be further addressed by analyzing the performance of the whole framework.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Digital Transformation in Industry · IoT and Edge/Fog Computing · Flexible and Reconfigurable Manufacturing Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Mechanical Engineering, Aalto University, 02150 Espoo, Finland
Funding
Business Finland
References (66)
Show source-provided references
1. On the way from Industry 4.0 to Industry 5.0: From digital manufacturing to digital society
2. Industry 4.0 and Industry 5.0—Inception, conception and perception10.1016/j.jmsy.2021.10.006 ↗
3. Nahavandi, S. (2019). Industry 5.0—A human-centric solution. Sustainability, 11.10.3390/su11164371 ↗
4. Industry 5.0: A survey on enabling technologies and potential applications
5. Guest editorial Industry 4.0–prerequisites and visions10.1109/TASE.2016.2523639 ↗
6. PwC (2022, March 28). Industry 4.0: Building the Digital Enterprise. 2016 Global Industry 4.0 Survey. Available online: https://www.pwc.com/gx/en/industries/industries-4.0/landing-page/industry-4.0-building-your-digital-enterprise-april-2016.pdf.
7. The expected contribution of Industry 4.0 technologies for industrial performance10.1016/j.ijpe.2018.08.019 ↗
8. Industry 4.0 technologies: Implementation patterns in manufacturing companies10.1016/j.ijpe.2019.01.004 ↗
9. Nayak, N.G., Dürr, F., and Rothermel, K. (2015, January 26–28). Software-defined environment for reconfigurable manufacturing systems. Proceedings of the 2015 5th International Conference on the Internet of Things (IOT), Seoul, Korea.10.1109/IOT.2015.7356556 ↗
10. Longo, F., Padovano, A., and Umbrello, S. (2020). Value-oriented and ethical technology engineering in industry 5.0: A human-centric perspective for the design of the factory of the future. Appl. Sci., 10.10.3390/app10124182 ↗
11. Fatima, Z., Tanveer, M.H., Zardari, S., Naz, L.F., Khadim, H., Ahmed, N., and Tahir, M. (2022). Production Plant and Warehouse Automation with IoT and Industry 5.0. Appl. Sci., 12.10.3390/app12042053 ↗
12. Kaasinen, E., Anttila, A.H., Heikkilä, P., Laarni, J., Koskinen, H., and Väätänen, A. (2022). Smooth and Resilient Human–Machine Teamwork as an Industry 5.0 Design Challenge. Sustainability, 14.10.3390/su14052773 ↗
13. Digital twin enhanced human-machine interaction in product lifecycle10.1016/j.procir.2019.04.330 ↗
14. Digital twin for human-machine interaction with convolutional neural network10.1080/0951192X.2021.1925966 ↗
15. Review of digital twin applications in manufacturing10.1016/j.compind.2019.103130 ↗
16. Boschert, S., and Rosen, R. (2016). Digital twin—The simulation aspect. Mechatronic Futures, Springer.10.1007/978-3-319-32156-1_5 ↗
17. Grieves, M., and Vickers, J. (2017). Digital twin: Mitigating unpredictable, undesirable emergent behavior in complex systems. Transdisciplinary Perspectives on Complex Systems, Springer.10.1007/978-3-319-38756-7_4 ↗
18. Glaessgen, E., and Stargel, D. (2012, January 23–26). The digital twin paradigm for future NASA and US Air Force vehicles. Proceedings of the 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference 20th AIAA/ASME/AHS Adaptive Structures Conference 14th AIAA, Honolulu, HI, USA.10.2514/6.2012-1818 ↗
19. Reid, J., and Rhodes, D. (2016). Digital system models: An investigation of the non-technical challenges and research needs. Conference on Systems Engineering Research, Systems Engineering Advancement Research Initiative, Massachusetts Institute of Technology.
20. Fera, M., Greco, A., Caterino, M., Gerbino, S., Caputo, F., Macchiaroli, R., and D’Amato, E. (2019). Towards digital twin implementation for assessing production line performance and balancing. Sensors, 20.10.3390/s20010097 ↗
21. BIM Information Integration Based VR Modeling in Digital Twins in Industry 5.0
22. Digital twin-driven product design, manufacturing and service with big data10.1007/s00170-017-0233-1 ↗
23. Digital twin data modeling with automationml and a communication methodology for data exchange10.1016/j.ifacol.2016.11.115 ↗
24. Virtual factory: Digital twin based integrated factory simulations10.1016/j.procir.2020.04.043 ↗
25. Pérez, L., Rodríguez-Jiménez, S., Rodríguez, N., Usamentiaga, R., and García, D.F. (2020). Digital twin and virtual reality based methodology for multi-robot manufacturing cell commissioning. Appl. Sci., 10.10.3390/app10103633 ↗
26. Twinbase: Open-source server software for the Digital Twin Web10.1109/ACCESS.2021.3119487 ↗
27. A feature-based framework for structuring industrial digital twins10.1109/ACCESS.2019.2950507 ↗
28. An overview of current technologies and emerging trends in factory automation10.1080/00207543.2018.1510558 ↗
29. A cyber-physical systems architecture for industry 4.0-based manufacturing systems10.1016/j.mfglet.2014.12.001 ↗
30. Gorecky, D., Schmitt, M., Loskyll, M., and Zühlke, D. (2014, January 27–30). Human-machine-interaction in the industry 4.0 era. Proceedings of the 2014 12th IEEE International Conference on Industrial Informatics (INDIN), Porto Alegre, Brazil.10.1109/INDIN.2014.6945523 ↗
31. Automated and adaptive geometry preparation for ar/vr-applications10.1115/1.4053327 ↗
32. Industrial Case Studies for Digital Transformation of Engineering Processes using the Virtual Reality Technology10.1016/j.procir.2020.01.082 ↗
33. Interactive Sensor Dashboard for Smart Manufacturing10.1016/j.procs.2022.01.204 ↗
34. Burghardt, A., Szybicki, D., Gierlak, P., Kurc, K., Pietruś, P., and Cygan, R. (2020). Programming of industrial robots using virtual reality and digital twins. Appl. Sci., 10.10.3390/app10020486 ↗
35. He, F., Ong, S.K., and Nee, A.Y. (2021). An Integrated Mobile Augmented Reality Digital Twin Monitoring System. Computers, 10.10.3390/computers10080099 ↗
36. Majewski, M., and Kacalak, W. (2016). Human-machine speech-based interfaces with augmented reality and interactive systems for controlling mobile cranes. International Conference on Interactive Collaborative Robotics, Springer.10.1007/978-3-319-43955-6_12 ↗
37. Lin, Z., Petzold, F., and Hsieh, S. (2020). 4D-BIM Based Real Time Augmented Reality Navigation System for Tower Crane Operation. Construction Research Congress 2020: Computer Applications, American Society of Civil Engineers.10.1061/9780784482865.088 ↗
38. Quandt, M., Beinke, T., Freitag, M., and Kölsch, C. (2018). Requirements for an Augmented Reality-Based Assistance System. International Conference on Dynamics in Logistics, Springer.10.1007/978-3-319-74225-0_45 ↗
39. Evaluating Mobile Crane Lift Operations Using an Interactive Virtual Reality System10.1061/(ASCE)CO.1943-7862.0002177 ↗
40. A framework for extended reality system development in manufacturing10.1109/ACCESS.2021.3056752 ↗
41. A Digital Twin-Driven and Conceptual Framework for Enabling Extended Reality Applications: A Case Study of a Brake Discs Manufacturer10.1016/j.procs.2022.01.389 ↗
42. Pereira, V., Matos, T., Rodrigues, R., Nóbrega, R., and Jacob, J. (2019, January 21–22). Extended reality framework for remote collaborative interactions in virtual environments. Proceedings of the 2019 International Conference on Graphics and Interaction (ICGI), Faro, Portugal.10.1109/ICGI47575.2019.8955025 ↗
43. Tu, X., Autiosalo, J., Jadid, A., Tammi, K., and Klinker, G. (2021). A Mixed Reality Interface for a Digital Twin Based Crane. Appl. Sci., 11.10.3390/app11209480 ↗
44. Yang, C. (2021). Framework for Virtual Reality Digital Services Leveraging Digital Twin-Based Crane. [Master’s Thesis, Aalto University].
45. Autiosalo, J. (2018, January 5–8). Platform for industrial internet and digital twin focused education, research, and innovation: Ilmatar the overhead crane. Proceedings of the 2018 IEEE 4th World Forum on Internet of Things (WF-IoT), Singapore.10.1109/WF-IoT.2018.8355217 ↗
46. Autiosalo, J., Ala-Laurinaho, R., Mattila, J., Valtonen, M., Peltoranta, V., and Tammi, K. (2021). Towards integrated digital twins for industrial products: Case study on an overhead crane. Appl. Sci., 11.10.3390/app11020683 ↗
47. Ala-Laurinaho, R., Autiosalo, J., and Tammi, K. (2020). Open Sensor Manager for IIoT. J. Sens. Actuator Netw., 9.10.3390/jsan9020030 ↗
48. Digital twin for CNC machine tool: Modeling and using strategy10.1007/s12652-018-0946-5 ↗
49. Selected reflections on formal modeling in Industry 4.010.1016/j.procs.2020.09.118 ↗
50. Koubâa, A. (2017). Robot Operating System (ROS), Springer.10.1007/978-3-319-54927-9 ↗
51. Wikipedia Contributors (2022, March 28). What Is ROS?. Available online: http://wiki.ros.org/ROS/Introduction.
52. Maruyama, Y., Kato, S., and Azumi, T. (2016, January 1–10). Exploring the performance of ROS2. Proceedings of the 13th International Conference on Embedded Software, Pittsburgh, PA, USA.10.1145/2968478.2968502 ↗
53. Takaya, K., Asai, T., Kroumov, V., and Smarandache, F. (2016, January 13–15). Simulation environment for mobile robots testing using ROS and Gazebo. Proceedings of the 2016 20th International Conference on System Theory, Control and Computing (ICSTCC), Sinaia, Romania.10.1109/ICSTCC.2016.7790647 ↗
54. OPC FOUNDATION (2022, March 28). Unified Architecture. Available online: https://opcfoundation.org/about/opc-technologies/opc-ua/.
55. OPC UA–service-oriented architecture for industrial applications
56. Mattila, J., Ala-Laurinaho, R., Autiosalo, J., Salminen, P., and Tammi, K. (2022). Using Digital Twin Documents to Control a Smart Factory: Simulation Approach with ROS, Gazebo, and Twinbase. Machines, 10.10.3390/machines10040225 ↗
57. Chao, Y. (2022, April 12). OPC-Unity-Client. Available online: https://github.com/talentyc/OPCUA-CLIENT-IN-UNITY.
58. Soni, D., and Makwana, A. (2017, January 6–8). A survey on mqtt: A protocol of internet of things (iot). Proceedings of the International Conference On Telecommunication, Power Analysis And Computing Techniques (ICTPACT-2017), Chennai, India.
59. Yang, C. (2022, April 12). OPC UA-MQTT Wrapper. Available online: https://github.com/talentyc/OPCUA-MQTT-GATEWAY-MQTT-UNITY-CLIENT.
60. Hietala, J. (2022, April 12). OPC-UA-GraphQL-Wrapper. Available online: https://github.com/AaltoIIC/OPC-UA-GraphQL-Wrapper.
61. Hietala, J. (2020). Real-Time Two-Way Data Transfer with a Digital Twin via Web Interface. [Master’s Thesis, Aalto University].
62. Critical questions for Facebook’s virtual reality: Data, power and the metaverse10.14763/2021.4.1610 ↗
63. Wikipedia Contributors (2022, March 28). OpenXR. Available online: https://en.wikipedia.org/wiki/OpenXR.
64. Data Link for the Creation of Digital Twins10.1109/ACCESS.2020.3045856 ↗
65. Ala-Laurinaho, R., Mattila, J., Autiosalo, J., Hietala, J., Laaki, H., and Tammi, K. (2022). Comparison of REST and GraphQL Interfaces for OPC UA. Computers, 11.10.3390/computers11050065 ↗
66. Realizing virtual reality learning environment for industry 4.010.1016/j.procir.2019.02.025 ↗