Integrating Virtual, Mixed, and Augmented Reality to Human–Robot Interaction Applications Using Game Engines: A Brief Review of Accessible Software Tools and Frameworks
Source abstract
Abstract
This article identifies and summarizes software tools and frameworks proposed in the Human–Robot Interaction (HRI) literature for developing extended reality (XR) experiences using game engines. This review includes primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can control or interact with real robotic platforms using devices that extend the user’s reality. The objective of this article is not to present an extensive list of applications and tools. Instead, we present recent, relevant, common, and accessible frameworks and software tools implemented in research articles published in high-impact robotics conferences and journals. For this, we searched papers published during a seven-years period between 2015 and 2022 in relevant databases for robotics (Science Direct, IEEE Xplore, ACM digital library, Springer Link, and Web of Science). Additionally, we present and classify the application context of the reviewed articles in four groups: social robotics, programming of industrial robots, teleoperation of industrial robots, and Human–Robot collaboration (HRC).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human-Automation Interaction and Safety · Robot Manipulation and Learning · Virtual Reality Applications and Impacts
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
National Institute of Advanced Industrial Science and Technology (AIST), Tokyo 135-0064, Japan
References (97)
Show source-provided references
1. COVID-19 and extended reality (XR)10.1080/13683500.2020.1798896 ↗
2. Doolani, S., Wessels, C., Kanal, V., Sevastopoulos, C., Jaiswal, A., Nambiappan, H., and Makedon, F. (2020). A review of extended reality (xr) technologies for manufacturing training. Technologies, 8.10.3390/technologies8040077 ↗
3. Connecting the CoppeliaSim robotics simulator to virtual reality10.1016/j.softx.2020.100426 ↗
4. Variable admittance control of a hand exoskeleton for virtual reality-based rehabilitation tasks
5. Nguyen, V.T., and Dang, T. (2017, January 9–13). Setting up Virtual Reality and Augmented Reality Learning Environment in Unity. Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), Recife, Brazil.10.1109/ISMAR-Adjunct.2017.97 ↗
6. Morse, C. (2021). Gaming Engines: Unity, Unreal, and Interactive 3D Spaces, Taylor & Francis.10.1080/24751448.2021.1967068 ↗
7. Bartneck, C., Soucy, M., Fleuret, K., and Sandoval, E.B. (September, January 31). The robot engine—Making the unity 3D game engine work for HRI. Proceedings of the 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Kobe, Japan.10.1109/ROMAN.2015.7333561 ↗
8. De Melo, M.S.P., da Silva Neto, J.G., Da Silva, P.J.L., Teixeira, J.M.X.N., and Teichrieb, V. (2019, January 28–31). Analysis and comparison of robotics 3d simulators. Proceedings of the 2019 21st Symposium on Virtual and Augmented Reality (SVR), Rio de Janeiro, Brazil.
9. Augmented reality-based guidance in product assembly and maintenance/repair perspective: A state of the art review on challenges and opportunities10.1016/j.eswa.2022.118983 ↗
10. Zhang, W., and Wang, Z. (2021). Theory and Practice of VR/AR in K-12 Science Education—A Systematic Review. Sustainability, 13.10.3390/su132212646 ↗
11. Ho, P.T., Albajez, J.A., Santolaria, J., and Yagüe-Fabra, J.A. (2022). Study of Augmented Reality Based Manufacturing for Further Integration of Quality Control 4.0: A Systematic Literature Review. Appl. Sci., 12.10.3390/app12041961 ↗
12. Boboc, R.G., Gîrbacia, F., and Butilă, E.V. (2020). The application of augmented reality in the automotive industry: A systematic literature review. Appl. Sci., 10.10.3390/app10124259 ↗
13. Review on existing VR/AR solutions in human–robot collaboration10.1016/j.procir.2020.05.259 ↗
14. Suzuki, R., Karim, A., Xia, T., Hedayati, H., and Marquardt, N. (May, January 30). Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces. Proceedings of the CHI Conference on Human Factors in Computing Systems 2022, New Orleans, LA, USA.10.1145/3491102.3517719 ↗
15. Costa, G.d.M., Petry, M.R., and Moreira, A.P. (2022). Augmented Reality for Human &;Robot Collaboration and Cooperation in Industrial Applications: A Systematic Literature Review. Sensors, 22.10.3390/s22072725 ↗
16. Framework for a closed-loop cooperative human Cyber-Physical System for the mining industry driven by VR and AR: MHCPS10.1016/j.cie.2022.108050 ↗
17. Scalable edge cloud platforms for IoT services10.1016/j.jnca.2020.102785 ↗
18. Cyber-Physical Systems10.1109/MC.2017.105 ↗
19. Digital Twins and Cyber–Physical Systems toward Smart Manufacturing and Industry 4.0: Correlation and Comparison10.1016/j.eng.2019.01.014 ↗
20. Maruyama, T., Ueshiba, T., Tada, M., Toda, H., Endo, Y., Domae, Y., Nakabo, Y., Mori, T., and Suita, K. (2021). Digital Twin-Driven Human Robot Collaboration Using a Digital Human. Sensors, 21.10.3390/s21248266 ↗
21. Resilient digital twins10.1007/s12599-021-00721-z ↗
22. Sepasgozar, S.M. (2021). Differentiating digital twin from digital shadow: Elucidating a paradigm shift to expedite a smart, sustainable built environment. Buildings, 11.10.3390/buildings11040151 ↗
23. Human–cyber–physical systems (HCPSs) in the context of new-generation intelligent manufacturing10.1016/j.eng.2019.07.015 ↗
24. Evaluating quality in human-robot interaction: A systematic search and classification of performance and human-centered factors, measures and metrics towards an industry 5.010.1016/j.jmsy.2022.04.007 ↗
25. Industry 5.0 and Society 5.0—Comparison, complementation and co-evolution10.1016/j.jmsy.2022.07.010 ↗
26. Eberly, D. (2006). 3D Game Engine Design: A Practical Approach to Real-Time Computer Graphics, CRC Press.
27. Gregory, J. (2018). Game Engine Architecture, AK Peters/CRC Press.10.1201/9781315267845 ↗
28. Dickson, P.E., Block, J.E., Echevarria, G.N., and Keenan, K.C. (2017, January 3–5). An experience-based comparison of unity and unreal for a stand-alone 3D game development course. Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, Bologna, Italy.10.1145/3059009.3059013 ↗
29. Juliani, A., Berges, V.P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., and Mattar, M. (2018). Unity: A general platform for intelligent agents. arXiv.
30. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation10.7326/M18-0850 ↗
31. Budgen, D., and Brereton, P. (2006). Performing systematic literature reviews in software engineering. In Proceedings of the International conference on Software engineering. Assoc. Comput. Mach., 1051–1052.10.1145/1134285.1134500 ↗
32. Kitchenham, B. (2004). Procedures for Performing Systematic Reviews, Keele University. Technical Report.
33. Guidelines for conducting systematic mapping studies in software engineering: An update10.1016/j.infsof.2015.03.007 ↗
34. Keele, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering, Elsevier. Technical Report, EBSE Technical Report.
35. On the reliability of mapping studies in software engineering10.1016/j.jss.2013.04.076 ↗
36. (2022, October 10). Rosbridge Suite. Available online: http://wiki.ros.org/rosbridge/_suite.
37. SIGVerse: A cloud-based VR platform for research on multimodal human-robot interaction10.3389/frobt.2021.549360 ↗
38. (2022, October 10). ROS Sharp. Available online: https://github.com/siemens/ros-sharp.
39. (2022, October 10). ROS TCP Connector. Available online: https://github.com/Unity-Technologies/ROS-TCP-Connector.
40. Babaians, E., Tamiz, M., Sarfi, Y., Mogoei, A., and Mehrabi, E. (2018, January 10). Ros2unity3d; high-performance plugin to interface ros with unity3d engine. Proceedings of the 2018 9th Conference on Artificial Intelligence and Robotics and 2nd Asia-Pacific International Symposium, Kish Island, Iran.10.1109/AIAR.2018.8769798 ↗
41. Coronado, E., and Venture, G. (2020). Towards IoT-Aided Human–Robot Interaction Using NEP and ROS: A Platform-Independent, Accessible and Distributed Approach. Sensors, 20.10.3390/s20051500 ↗
42. (2022, October 10). ZeroMQ Socket Api. Available online: https://zeromq.org/socket-api/.
43. (2022, October 10). Photon Engine. Available online: https://www.photonengine.com/.
44. (2022, October 10). Mirror Networking. Available online: https://mirror-networking.gitbook.io/docs/.
45. (2022, October 10). Netcode for GameObjects. Available online: https://docs-multiplayer.unity3d.com/.
46. Operator support in human–robot collaborative environments using AI enhanced wearable devices10.1016/j.procir.2020.07.006 ↗
47. Virtual reality environment for industrial robot control and path design10.1016/j.procir.2021.05.021 ↗
48. Gao, Y., and Huang, C.M. (2019, January 17–20). PATI: A projection-based augmented table-top interface for robot programming. Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, CA, USA.10.1145/3301275.3302326 ↗
49. (2022, October 10). TouchScript. Available online: https://github.com/TouchScript/TouchScript.
50. Tutorial: Point cloud library: Three-dimensional object recognition and 6 dof pose estimation10.1109/MRA.2012.2206675 ↗
51. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography10.1145/358669.358692 ↗
52. Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction10.1016/j.aei.2020.101170 ↗
53. (2022, October 10). PointNet. Available online: https://github.com/charlesq34/pointnet.
54. (2022, October 10). Vuforia Engine Package Unity. Available online: https://library.vuforia.com/getting-started/vuforia-engine-package-unity.
55. An augmented reality framework for robotic tool-path teaching10.1016/j.procir.2020.03.143 ↗
56. (2022, October 10). Google ARCore. Available online: https://developers.google.com/ar.
57. (2022, October 10). Mixed Reality Toolkit. Available online: https://github.com/microsoft/MixedRealityToolkit-Unity.
58. AR based robot programming using teaching by demonstration techniques10.1016/j.procir.2020.09.186 ↗
59. (2022, October 10). Zxing. Available online: https://github.com/zxing/zxing.
60. Botev, J., and Rodríguez Lera, F.J. (2021). Immersive robotic telepresence for remote educational scenarios. Sustainability, 13.10.3390/su13094717 ↗
61. (2023, January 01). IAI Kinect. Available online: https://nuitrack.com/.
62. OpenCV
63. (2022, October 10). Find Object 2D ROS package. Available online: http://wiki.ros.org/find_object_2d.
64. Moveit (2022, October 10). Motion Planning Framework, Available online: https://moveit.ros.org/.
65. (2023, January 01). IAI Kinect. Available online: https://github.com/code-iai/iai_kinect2.
66. (2023, January 01). RobCog-IAI. Available online: https://github.com/robcog-iai.
67. (2022, October 10). Newton VR. Available online: https://assetstore.unity.com/packages/tools/newtonvr-75712.
68. Li, R., van Almkerk, M., van Waveren, S., Carter, E., and Leite, I. (2019, January 11–14). Comparing human-robot proxemics between virtual reality and the real world. Proceedings of the 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Daegu, Korea.10.1109/HRI.2019.8673116 ↗
69. Exploiting virtual reality and the robot operating system to remote-control a humanoid robot10.1007/s11042-022-12021-z ↗
70. Shariati, A., Shahab, M., Meghdari, A., Amoozandeh Nobaveh, A., Rafatnejad, R., and Mozafari, B. Virtual reality social robot platform: A case study on Arash social robot. Proceedings of the International Conference on Social Robotics.
71. Pot, E., Monceaux, J., Gelin, R., and Maisonnier, B. (2009, January 14–18). Choregraphe: A graphical tool for humanoid robot programming. Proceedings of the RO-MAN 2009-The 18th IEEE International Symposium on Robot and Human Interactive Communication, New Delhi, India.10.1109/ROMAN.2009.5326209 ↗
72. Visual programming environments for end-user development of intelligent and social robots, a systematic review10.1016/j.cola.2020.100970 ↗
73. Cao, Y., Wang, T., Qian, X., Rao, P.S., Wadhawan, M., Huo, K., and Ramani, K. (2019, January 20–23). GhostAR: A time-space editor for embodied authoring of human-robot collaborative task with augmented reality. Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, New Orleans, LA, USA.10.1145/3332165.3347902 ↗
74. Ostanin, M., Mikhel, S., Evlampiev, A., Skvortsova, V., and Klimchik, A. (August, January 31). Human-robot interaction for robotic manipulator programming in Mixed Reality. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9196965 ↗
75. Soares, I., Petry, M., and Moreira, A.P. (2021). Programming Robots by Demonstration Using Augmented Reality. Sensors, 21.10.3390/s21175976 ↗
76. Karan, M.S., Berkman, M.İ., and Çatak, G. (2021). Smartphone as a Paired Game Input Device: An Application on HoloLens Head Mounted Augmented Reality System. Game+ Design Education, Springer.10.1007/978-3-030-65060-5_22 ↗
77. Augmented reality and ultrasound education: Initial experience10.1053/j.jvca.2017.12.006 ↗
78. Tian-Han, G., Qiao-Yu, T., and Shuo, Z. (2018, January 20–22). The virtual museum based on HoloLens and vuforia. Proceedings of the 2018 4th Annual International Conference on Network and Information Systems for Computers (ICNISC), Wuhan, China.10.1109/ICNISC.2018.00084 ↗
79. Teleoperation of industrial robot manipulators based on augmented reality10.1007/s00170-020-05997-1 ↗
80. The Vicarios Virtual Reality Interface for Remote Robotic Teleoperation10.1007/s10846-021-01311-7 ↗
81. Mixed reality-integrated 3D/2D vision mapping for intuitive teleoperation of mobile manipulator10.1016/j.rcim.2022.102332 ↗
82. Whitney, D., Rosen, E., Ullman, D., Phillips, E., and Tellex, S. Ros reality: A virtual reality framework using consumer-grade hardware for ros-enabled robots. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593513 ↗
83. Designing an AR interface to improve trust in Human-Robots collaboration10.1016/j.procir.2018.01.009 ↗
84. Closed-loop augmented reality towards accurate human-robot collaboration10.1016/j.cirp.2020.03.014 ↗
85. Mahadevan, K., Sousa, M., Tang, A., and Grossman, T. (2021, January 8–13). “grip-that-there”: An investigation of explicit and implicit task allocation techniques for human-robot collaboration. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.10.1145/3411764.3445355 ↗
86. Chandan, K., Kudalkar, V., Li, X., and Zhang, S. (June, January 30). ARROCH: Augmented reality for robots collaborating with a human. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9561144 ↗
87. Weber, D., Kasneci, E., and Zell, A. (2022, January 7–10). Exploiting Augmented Reality for Extrinsic Robot Calibration and Eye-based Human-Robot Collaboration. Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction, Sapporo, Hokkaido, Japan.10.1109/HRI53351.2022.9889538 ↗
88. Digital twin-enabled advance execution for human-robot collaborative assembly10.1016/j.cirp.2022.03.024 ↗
89. Human motion quality and accuracy measuring method for human–robot physical interactions10.1007/s11370-022-00432-8 ↗
90. An augmented reality software suite enabling seamless human robot interaction
91. Augmented reality (ar) based framework for supporting human workers in flexible manufacturing10.1016/j.procir.2021.01.091 ↗
92. Digital twins for collaborative robots: A case study in human-robot interaction10.1016/j.rcim.2020.102092 ↗
93. Function block-based human-robot collaborative assembly driven by brainwaves10.1016/j.cirp.2021.04.091 ↗
94. Estimating cybersickness from virtual reality applications10.1007/s10055-020-00446-6 ↗
95. Exploration of two safety strategies in human-robot collaborative manufacturing using Virtual Reality10.1016/j.promfg.2020.01.066 ↗
96. Montalvo, W., Bonilla-Vasconez, P., Altamirano, S., Garcia, C.A., and Garcia, M.V. (2020, January 7–10). Industrial Control Robot Based on Augmented Reality and IoT Protocol. Proceedings of the International Conference on Augmented Reality, Virtual Reality and Computer Graphics, Virtual Event.10.1007/978-3-030-58468-9_25 ↗
97. Botev, J., and Rodríguez Lera, F.J. (2020). Immersive Telepresence Framework for Remote Educational Scenarios. Proceedings of the International Conference on Human-Computer Interaction, Springer.10.3390/su13094717 ↗