A Systematic Review of Virtual Reality Interfaces for Controlling and Interacting with Robots
Source abstract
Abstract
There is a significant amount of synergy between virtual reality (VR) and the field of robotics. However, it has only been in approximately the past five years that commercial immersive VR devices have been available to developers. This new availability has led to a rapid increase in research using VR devices in the field of robotics, especially in the development of VR interfaces for operating robots. In this paper, we present a systematic review on VR interfaces for robot operation that utilize commercially available immersive VR devices. A total of 41 papers published between 2016–2020 were collected for review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Papers are discussed and categorized into five categories: (1) Visualization, which focuses on displaying data or information to operators; (2) Robot Control and Planning, which focuses on connecting human input or movement to robot movement; (3) Interaction, which focuses on the development of new interaction techniques and/or identifying best interaction practices; (4) Usability, which focuses on user experiences of VR interfaces; and (5) Infrastructure, which focuses on system architectures or software to support connecting VR and robots for interface development. Additionally, we provide future directions to continue development in VR interfaces for operating robots.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Virtual Reality Applications and Impacts · Augmented Reality Applications · Teleoperation and Haptic Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute for Experiential Robotics, Northeastern University, Boston, MA 02115, USA
Funding
National Aeronautics and Space Administration · National Science Foundation
References (60)
Show source-provided references
1. Mazuryk, T., and Gervautz, M. (1996). Virtual Reality-History, Applications, Technology and Future, Vienna University of Technology.
2. Current state of virtual reality simulation in robotic surgery training: A review10.1007/s00464-015-4517-y ↗
3. A systematic review of virtual reality simulators for robot-assisted surgery10.1016/j.eururo.2015.09.021 ↗
4. The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: A current review10.1007/s00464-008-0298-x ↗
5. Sensorimotor training in virtual reality: A review
6. Trends in robot-assisted and virtual reality-assisted neuromuscular therapy: A systematic review of health-related multiplayer games10.1186/s12984-018-0449-9 ↗
7. A meta-analysis and systematic literature review of virtual reality rehabilitation programs10.1016/j.chb.2017.01.013 ↗
8. Invited review: The synergy between virtual reality and robotics10.1109/70.768174 ↗
9. HapticSnakes: Multi-haptic feedback wearable robots for immersive virtual reality10.1007/s10055-019-00404-x ↗
10. Vonach, E., Gatterer, C., and Kaufmann, H. (2017, January 18–22). VRRobot: Robot actuated props in an infinite virtual environment. Proceedings of the 2017 IEEE Virtual Reality (VR), Los Angeles, CA, USA.10.1109/VR.2017.7892233 ↗
11. Zhao, Y., Kim, L.H., Wang, Y., Le Goc, M., and Follmer, S. (2017, January 17–20). Robotic Assembly of Haptic Proxy Objects for Tangible Interaction and Virtual Reality. Proceedings of the Interactive Surfaces and Spaces on ZZZ—ISS ’17, Brighton, UK.10.1145/3132272.3134143 ↗
12. Suzuki, R., Hedayati, H., Zheng, C., Bohn, J.L., Szafir, D., Do, E.Y.L., Gross, M.D., and Leithinger, D. (2020, January 25–30). RoomShift: Room-Scale Dynamic Haptics for VR with Furniture-Moving Swarm Robots. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, Honolulu, HI, USA.10.1145/3313831.3376523 ↗
13. Bi, Y., Bhatia, R., and Kapoor, S. (2020). Telexistence and Teleoperation for Walking Humanoid Robots. Intelligent Systems and Applications, Springer International Publishing.
14. Bolano, G., Roennau, A., Dillmann, R., and Groz, A. (2020, January 22–26). Virtual Reality for Offline Programming of Robotic Applications with Online Teaching Methods. Proceedings of the 2020 17th International Conference on Ubiquitous Robots (UR), Kyoto, Japan.10.1109/UR49135.2020.9144806 ↗
15. Liu, O., Rakita, D., Mutlu, B., and Gleicher, M. (September, January 28). Understanding human-robot interaction in virtual reality. Proceedings of the 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Lisbon, Portugal.10.1109/ROMAN.2017.8172387 ↗
16. Villani, V., Capelli, B., and Sabattini, L. (2018, January 27–31). Use of Virtual Reality for the Evaluation of Human-Robot Interaction Systems in Complex Scenarios. Proceedings of the 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Nanjing, China.10.1109/ROMAN.2018.8525738 ↗
17. Wijnen, L., Lemaignan, S., and Bremner, P. (2020, January 23–26). Towards using Virtual Reality for Replicating HRI Studies. Proceedings of the Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’20, Cambridge, UK.10.1145/3371382.3378374 ↗
18. Design of a virtual reality training system for human–robot collaboration in manufacturing tasks10.1007/s12008-015-0259-2 ↗
19. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., and Prisma Group (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med., 6.10.1371/journal.pmed.1000097 ↗
20. Chen, J.Y.C., and Fragomeni, G. (2019). Human-Robot Interaction During Virtual Reality Mediated Teleoperation: How Environment Information Affects Spatial Task Performance and Operator Situation Awareness. Virtual, Augmented and Mixed Reality, Applications and Case Studies, Springer International Publishing.
21. Su, Y.H., Xu, Y.Q., Cheng, S.L., Ko, C.H., and Young, K.Y. (2019, January 9–12). Development of an Effective 3D VR-Based Manipulation System for Industrial Robot Manipulators. Proceedings of the 2019 12th Asian Control Conference (ASCC), Kitakyushu-shi, Japan.
22. Kohn, S., Blank, A., Puljiz, D., Zenkel, L., Bieber, O., Hein, B., and Franke, J. (2018, January 1–5). Towards a Real-Time Environment Reconstruction for VR-Based Teleoperation Through Model Segmentation. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8594053 ↗
23. Effects of Augmented Reality on the Performance of Teleoperated Industrial Assembly Tasks in a Robotic Embodiment10.1109/THMS.2017.2782490 ↗
24. Stotko, P., Krumpen, S., Schwarz, M., Lenz, C., Behnke, S., Klein, R., and Weinmann, M. (2019, January 3–8). A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot. Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.10.1109/IROS40897.2019.8968598 ↗
25. Xinxing, T., Pengfei, Z., and Hironao, Y. (2016, January 28–30). VR-based construction tele-robot system displayed by HMD with active viewpoint movement mode. Proceedings of the 2016 Chinese Control and Decision Conference (CCDC), Yinchuan, China.10.1109/CCDC.2016.7532231 ↗
26. Head-synced Drone Control for Reducing Virtual Reality Sickness10.1007/s10846-019-01054-6 ↗
27. Zhao, J., Allison, R.S., Vinnikov, M., and Jennings, S. (2018, January 7–10). The Effects of Visual and Control Latency on Piloting a Quadcopter Using a Head-Mounted Display. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.10.1109/SMC.2018.00505 ↗
28. Salichs, M.A., Ge, S.S., Barakova, E.I., Cabibihan, J.J., Wagner, A.R., Castro-González, Á., and He, H. (2019). Improving the Visual Comfort of Virtual Reality Telepresence for Robotics. Social Robotics, Springer International Publishing.
29. Theofilis, K., Orlosky, J., Nagai, Y., and Kiyokawa, K. (2016, January 15–17). Panoramic view reconstruction for stereoscopic teleoperation of a humanoid robot. Proceedings of the 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), Cancun, Mexico.10.1109/HUMANOIDS.2016.7803284 ↗
30. Application of virtual reality in teleoperation of the military mobile robotic system TAROS10.1177/1729881417751545 ↗
31. Human Perception-Optimized Planning for Comfortable VR-Based Telepresence10.1109/LRA.2020.3015191 ↗
32. Baxter’s Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing10.1109/LRA.2017.2737046 ↗
33. Hirschmanner, M., Tsiourti, C., Patten, T., and Vincze, M. (2019, January 15–17). Virtual Reality Teleoperation of a Humanoid Robot Using Markerless Human Upper Body Pose Imitation. Proceedings of the 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), Toronto, ON, Canada.10.1109/Humanoids43949.2019.9035064 ↗
34. Gaurav, S., Al-Qurashi, Z., Barapatre, A., Maratos, G., Sarma, T., and Ziebart, B.D. (2019, January 15–17). Deep Correspondence Learning for Effective Robotic Teleoperation Using Virtual Reality. Proceedings of the 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), Toronto, ON, Canada.10.1109/Humanoids43949.2019.9035031 ↗
35. Kamali, K., Bonev, I.A., and Desrosiers, C. (2020, January 13–15). Real-time Motion Planning for Robotic Teleoperation Using Dynamic-goal Deep Reinforcement Learning. Proceedings of the 2020 17th Conference on Computer and Robot Vision (CRV), Ottawa, ON, Canada.10.1109/CRV50864.2020.00032 ↗
36. Xi, B., Wang, S., Ye, X., Cai, Y., Lu, T., and Wang, R. (2019). A robotic shared control teleoperation method based on learning from demonstrations. Int. J. Adv. Robot. Syst., 16.10.1177/1729881419857428 ↗
37. Tsokalo, I.A., Kuss, D., Kharabet, I., Fitzek, F.H.P., and Reisslein, M. (2019, January 9–13). Remote Robot Control with Human-in-the-Loop over Long Distances Using Digital Twins. Proceedings of the 2019 IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA.10.1109/GLOBECOM38437.2019.9013428 ↗
38. A New Mixed-Reality-Based Teleoperation System for Telepresence and Maneuverability Enhancement10.1109/THMS.2019.2960676 ↗
39. Amato, N.M., Hager, G., Thomas, S., and Torres-Torriti, M. (2020). Comparing Robot Grasping Teleoperation Across Desktop and Virtual Reality with ROS Reality. Robotics Research, Springer International Publishing.10.1007/978-3-030-28619-4 ↗
40. Just, C., Ortmaier, T., and Kahrs, L.A. (2018, January 20–21). A user study on robot path planning inside a Virtual Reality environment. Proceedings of the ISR 2018 50th International Symposium on Robotics, Munich, Germany.
41. Virtual Reality Robot-Assisted Welding Based on Human Intention Recognition10.1109/TASE.2019.2945607 ↗
42. A Virtual Reality Interface for an Autonomous Spray Painting UAV10.1109/LRA.2019.2922588 ↗
43. Yashin, G.A., Trinitatova, D., Agishev, R.T., Ibrahimov, R., and Tsetserukou, D. (2019, January 2–6). AeroVr: Virtual Reality-based Teleoperation with Tactile Feedback for Aerial Manipulation. Proceedings of the 2019 19th International Conference on Advanced Robotics (ICAR), Belo Horizonte, Brazil.10.1109/ICAR46387.2019.8981574 ↗
44. Gorjup, G., Dwivedi, A., Elangovan, N., and Liarokapis, M. (2019, January 3–8). An Intuitive, Affordances Oriented Telemanipulation Framework for a Dual Robot Arm Hand System: On the Execution of Bimanual Tasks. Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.10.1109/IROS40897.2019.8967782 ↗
45. Lee, J., Lu, Y., Xu, Y., and Song, D. (2016, January 9–14). Visual programming for mobile robot navigation using high-level landmarks. Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea.10.1109/IROS.2016.7759449 ↗
46. Roldán, J., Peña-Tapia, E., Martín-Barrio, A., Olivares-Méndez, M., Del Cerro, J., and Barrientos, A. (2017). Multi-Robot Interfaces and Operator Situational Awareness: Study of the Impact of Immersion and Prediction. Sensors, 17.10.3390/s17081720 ↗
47. Du, J., Sheng, W., and Liu, M. (2016, January 9–14). Human-guided robot 3D mapping using virtual reality technology. Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea.10.1109/IROS.2016.7759680 ↗
48. Hetrick, R., Amerson, N., Kim, B., Rosen, E., de Visser, E.J., and Phillips, E. (2020, January 24). Comparing Virtual Reality Interfaces for the Teleoperation of Robots. Proceedings of the 2020 Systems and Information Engineering Design Symposium (SIEDS), Charlottesville, VA, USA.10.1109/SIEDS49339.2020.9106630 ↗
49. Chen, J.Y.C., and Fragomeni, G. (2019). Design of Virtual Reality for Humanoid Robots with Inspiration from Video Games. Virtual, Augmented and Mixed Reality, Applications and Case Studies, Springer International Publishing.
50. Franzluebbers, A., and Johnson, K. (2019, January 19–20). Remote Robotic Arm Teleoperation through Virtual Reality. Proceedings of the Symposium on Spatial User Interaction, SUI ’19, New Orleans, LA, USA.10.1145/3357251.3359444 ↗
51. Theofanidis, M., Sayed, S.I., Lioulemes, A., and Makedon, F. (2017, January 21–23). VARM. Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA ’17, Island of Rhodes, Greece.10.1145/3056540.3056541 ↗
52. Szewczyk, R., Zieliński, C., and Kaliczyńska, M. (2020). Measuring Performance in Robotic Teleoperation Tasks with Virtual Reality Headgear. Automation 2019, Springer International Publishing.10.1007/978-3-030-13273-6 ↗
53. Bringing Adaptive and Immersive Interfaces to Real-World Multi-Robot Scenarios: Application to Surveillance and Intervention in Infrastructures10.1109/ACCESS.2019.2924938 ↗
54. Conn, M.A., and Sharma, S. (November, January 31). Immersive Telerobotics Using the Oculus Rift and the 5DT Ultra Data Glove. Proceedings of the 2016 International Conference on Collaboration Technologies and Systems (CTS), Orlando, FL, USA.10.1109/CTS.2016.0075 ↗
55. Koubaa, A. (2019). Multi-robot Systems, Virtual Reality and ROS: Developing a New Generation of Operator Interfaces. Robot Operating System (ROS): The Complete Reference (Volume 3), Springer International Publishing.
56. Mizuchi, Y., and Inamura, T. (2017, January 11–14). Cloud-based multimodal human-robot interaction simulator utilizing ROS and unity frameworks. Proceedings of the 2017 IEEE/SICE International Symposium on System Integration (SII), Taipei, Taiwan.10.1109/SII.2017.8279345 ↗
57. Whitney, D., Rosen, E., Ullman, D., Phillips, E., and Tellex, S. (2018, January 1–5). ROS Reality: A Virtual Reality Framework Using Consumer-Grade Hardware for ROS-Enabled Robots. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593513 ↗
58. Bischoff, M. (2020, December 17). ROS#. Available online: https://github.com/siemens/ros-sharp.
59. Babaians, E., Tamiz, M., Sarfi, Y., Mogoei, A., and Mehrabi, E. (2018, January 10). ROS2Unity3D; High-Performance Plugin to Interface ROS with Unity3d engine. Proceedings of the 2018 9th Conference on Artificial Intelligence and Robotics and 2nd Asia-Pacific International Symposium, Kish Island, Iran.10.1109/AIAR.2018.8769798 ↗
60. Astad, M.A., Hauan Arbo, M., Grotli, E.I., and Tommy Gravdahl, J. (2019, January 6–8). Vive for Robotics: Rapid Robot Cell Calibration. Proceedings of the 2019 7th International Conference on Control, Mechatronics and Automation (ICCMA), Delft, The Netherlands.10.1109/ICCMA46720.2019.8988631 ↗