Interface Transparency Issues in Teleoperation
Source abstract
Abstract
Transferring skills and expertise to remote places, without being present, is a new challenge for our digitally interconnected society. People can experience and perform actions in distant places through a robotic agent wearing immersive interfaces to feel physically there. However, technological contingencies can affect human perception, compromising skill-based performances. Considering the results from studies on human factors, a set of recommendations for the construction of immersive teleoperation systems is provided, followed by an example of the evaluation methodology. We developed a testbed to study perceptual issues that affect task performance while users manipulated the environment either through traditional or immersive interfaces. The analysis of its effect on perception, navigation, and manipulation relies on performances measures and subjective answers. The goal is to mitigate the effect of factors such as system latency, field of view, frame of reference, or frame rate to achieve the sense of telepresence. By decoupling the flows of an immersive teleoperation system, we aim to understand how vision and interaction fidelity affects spatial cognition. Results show that misalignments between the frame of reference for vision and motor-action or the use of tools affecting the sense of body position or movement have a higher effect on mental workload and spatial cognition.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Virtual Reality Applications and Impacts · Tactile and Sensory Interactions · Teleoperation and Haptic Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Ci2—Smart Cities Research Center, Polytechnic Institute of Tomar, 2300-313 Tomar, Portugal
University of Coimbra, Institute of Systems and Robotics, 3030-290 Coimbra, Portugal
Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University of Science and Technology (KU), Abu Dhabi 127788, UAE
Funding
OE - National funds of FCT/MCTES (PIDDAC)
References (100+)
Show source-provided references
1. MTR Corporation, Hong Kong (2020, August 28). MTR Deploys New “Vapourised Hydrogen Peroxide Robot” to Further Enhance Disinfection of Stations and Trains. Available online: https://www.mtr.com.hk/archive/corporate/en/press_release/PR-20-020-E.pdf.
2. Human-in-the-loop Control of a Humanoid Robot for Disaster Response: A Report from the DARPA Robotics Challenge Trials10.1002/rob.21567 ↗
3. Murphy, R. (2020, August 28). Real Robots to Help Fight Ebola (spectrum.ieee.org news). Available online: http://spectrum.ieee.org/automaton/robotics/medical-robots/real-robots-to-help-fight-ebola.
4. Introduction to Special Section on Surgical Robotics10.1109/TBME.2013.2242671 ↗
5. Baker, W., Kingston, Z., Moll, M., Badger, J., and Kavraki, L.E. (2017, January 8–10). Robonaut 2 and you: Specifying and executing complex operations. Proceedings of the 2017 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), Austin, TX, USA.10.1109/ARSO.2017.8025204 ↗
6. Two years of Visual Odometry on the Mars Exploration Rovers10.1002/rob.20184 ↗
7. Design Challenges and Guidelines for Social Interaction Using Mobile Telepresence Robots10.1177/1557234X13502462 ↗
8. Human Performance Issues and User Interface Design for Teleoperated Robots10.1109/TSMCC.2007.905819 ↗
9. Merging the senses into a robust percept10.1016/j.tics.2004.02.002 ↗
10. Sheridan, T.B. (1992). Telerobotics, Automation, and Human Supervisory Control, MIT Press.
11. Lombard, M., Biocca, F., Freeman, J., IJsselsteijn, W., and Schaevitz, R. (2015). Immersed in Media: Telepresence Theory, Measurement & Technology, Springer International Publishing.10.1007/978-3-319-10190-3 ↗
12. Virtual reality in neuroscience research and therapy10.1038/nrn3122 ↗
13. Musings on telepresence and virtual presence10.1162/pres.1992.1.1.120 ↗
14. Telepresence
15. Almeida, L., Patrao, B., Menezes, P., and Dias, J. (2014, January 25–29). Be the robot: Human embodiment in tele-operation driving tasks. Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN, 2014), Edinburgh, UK.10.1109/ROMAN.2014.6926298 ↗
16. How we experience immersive virtual environments: the concept of presence and its measurement
17. A Natural Interface for Remote Operation of Underwater Robots10.1109/MCG.2015.118 ↗
18. Martins, H., and Ventura, R. (2009, January 22–25). Immersive 3-d teleoperation of a search and rescue robot using a head-mounted display. Proceedings of the 2009 IEEE Conference on Emerging Technologies & Factory Automation, Palma de Mallorca, Spain.10.1109/ETFA.2009.5347014 ↗
19. Managing workload in human–robot interaction: A review of empirical studies10.1016/j.chb.2010.03.010 ↗
20. Performance evaluation of a medical robotic 3D-ultrasound imaging system10.1016/j.media.2007.10.006 ↗
21. Limited Field of View of Head-mounted Displays is Not the Cause of Distance Underestimation in Virtual Environments10.1162/1054746042545238 ↗
22. The effect of visual and interaction fidelity on spatial cognition in immersive virtual environments10.1109/TVCG.2006.55 ↗
23. Experimental investigation of the relationship between subjective telepresence and performance in hand-eye tasks10.1117/12.197333 ↗
24. Telexistence: Enabling Humans to Be Virtually Ubiquitous10.1109/MCG.2016.6 ↗
25. Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models10.1109/TSMC.1983.6313160 ↗
26. Two hands, one brain: cognitive neuroscience of bimanual skill10.1016/j.tics.2003.10.017 ↗
27. Investigating users’ intuitive interaction with complex artefacts10.1016/j.apergo.2009.04.010 ↗
28. Sense of agency in health and disease: a review of cue integration approaches10.1016/j.concog.2011.08.010 ↗
29. Full-body illusions and minimal phenomenal selfhood10.1016/j.tics.2008.10.003 ↗
30. Goldstein, E., and Brockmole, J. (2016). Sensation and Perception, Cengage Learning.
31. Damasio, A. (1999). The Feeling of what Happens: Body and Emotion in the Making of Consciousness, Harvest Book; Harcourt Brace.
32. Metzinger, T. (2003). Being No One: The Self-Model Theory of Subjectivity, MIT Press.10.7551/mitpress/1551.001.0001 ↗
33. Amin, M.S. (2020, August 28). Vestibuloocular Reflex Testing; Medscape; 2016. Available online: https://emedicine.medscape.com/article/1836134-overview.
34. Mania, K., Adelstein, B.D., Ellis, S.R., and Hill, M.I. (2004, January 7–8). Perceptual Sensitivity to Head Tracking Latency in Virtual Environments with Varying Degrees of Scene Complexity. Proceedings of the 1st Symposium on Applied Perception in Graphics and Visualization, Los Angeles, CA, USA.10.1145/1012551.1012559 ↗
35. Visual acuity measurement standard
36. Resolution Acuity is better than Vernier Acuity10.1016/S0042-6989(97)85878-6 ↗
37. The history of the Ophthalmological Society of Copenhagen 1900–5010.1034/j.1600-0420.80.s234.2.x ↗
38. Cuervo, E., Chintalapudi, K., and Kotaru, M. (2018, January 24). Creating the Perfect Illusion: What Will It Take to Create Life-Like Virtual Reality Headsets?. Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications, Tempe, AZ, USA.10.1145/3177102.3177115 ↗
39. Judd, D.B., and Wyszecki, G. (1975). Color in Business, Science, and Industry, Wiley. [3rd ed.].
40. Evaluation of color encodings for high dynamic range pixels
41. Concurrent Performance of Gunner’s and Robotics Operator’s Tasks in a Multitasking Environment10.1080/08995600802565785 ↗
42. Darken, R.P., and Cevik, H. (1999, January 13–17). Map usage in virtual environments: orientation issues. Proceedings of the IEEE Virtual Reality (Cat. No. 99CB36316), Houston, TX, USA.
43. Multiple Two-Dimensional Displays as an Alternative to Three-Dimensional Displays in Telerobotic Tasks10.1518/001872000779698060 ↗
44. Karwowski, W. (2006). Design of visual displays for teleoperation. International Encyclopedia of Ergonomics and Human Factors, CRC Press Inc.. [2nd ed.].
45. Display Signaling in Augmented Reality: Effects of Cue Reliability and Image Realism on Attention Allocation and Trust Calibration10.1518/001872001775898269 ↗
46. Folds, D.J., and Gerth, J.M. (1994, January 24–28). Auditory Monitoring of up to Eight Simultaneous Sources. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Nashville, TN, USA.10.1177/154193129403800910 ↗
47. Retinal limits to the detection and resolution of gratings10.1364/JOSAA.4.001524 ↗
48. Hyperacuity thresholds of 1 sec: theoretical predictions and empirical validation10.1364/JOSAA.2.001170 ↗
49. Peripheral vision and pattern recognition: A review10.1167/11.5.13 ↗
50. LaValle, S. (2019). Virtual Reality, Cambridge University Press.
51. Ross, B., Bares, J., Stager, D., Jackel, L., and Perschbacher, M. (2007). An Advanced Teleoperation Testbed. 6th International Conference on Field and Service Robotics-FSR 2007, Springer.
52. An Evaluation of Level of Detail Degradation in Head-mounted Display Peripheries10.1162/pres.1997.6.6.630 ↗
53. Clinical application of a modern high-definition head-mounted display in sonography10.7863/ultra.33.8.1499 ↗
54. Surgical Performance with Head-Mounted Displays in Laparoscopic Surgery10.1089/lap.2008.0142.supp ↗
55. Teleoperator performance with varying force and visual feedback10.1177/001872089403600109 ↗
56. Human–Robot Interaction in the Context of Simulated Route Reconnaissance Missions10.1080/08995600802115904 ↗
57. Reaching for objects in VR displays: lag and frame rate10.1145/198425.198426 ↗
58. Meehan, M., Insko, B., Whitton, M., and Brooks, F. (2002). Physiological Measures of Presence in Stressful Virtual Environments. ACM Trans. Graph., 21.10.1145/566654.566630 ↗
59. Claypool, K.T., and Claypool, M. (2007). The effects of resolution on users playing first person shooter games. Multimedia Computing and Networking, SPIE.10.1117/12.705995 ↗
60. Review of Low Frame Rate Effects on Human Performance10.1109/TSMCA.2007.904779 ↗
61. Rift, O. (2020, August 28). PC SDK Developer Guide, Guidelines for VR Performance Optimization. Available online: https://developer.oculus.com/documentation/native/pc/dg-performance-guidelines/.
62. From presence to consciousness through virtual reality10.1038/nrn1651 ↗
63. Human space exploration and human spaceflight: Latency and the cognitive scale of the universe10.1016/j.spacepol.2011.02.002 ↗
64. Meehan, M., Razzaque, S., Whitton, M.C., and Brooks, F.P. (2003). Effect of latency on presence in stressful virtual environments. IEEE Virtual Reality 2003, Proceedings, IEEE.10.1109/VR.2003.1191132 ↗
65. Generalizeability of latency detection in a variety of virtual environments10.1177/154193120404802306 ↗
66. Saccadic Suppression of Retinotopically Localized Blood Oxygen Level-Dependent Responses in Human Primary Visual Area V110.1523/JNEUROSCI.0817-06.2006 ↗
67. Latency Requirements for Foveated Rendering in Virtual Reality10.1145/3127589 ↗
68. Towards Virtual Reality Infinite Walking: Dynamic Saccadic Redirection10.1145/3197517.3201294 ↗
69. Latency requirements for head-worn display S/EVS applications10.1117/12.554462 ↗
70. Rift Oculus (2020, August 28). PC SDK Developer Guide. Available online: https://developer.oculus.com/design/bp-rendering/.
71. Rift Oculus (2020, August 28). Oculus Rift CV1. Available online: https://www.oculus.com/rift.
72. (2020, August 28). HTC VIVE. Available online: https://www.vive.com/eu/.
73. Lane, J.C., Carignan, C.R., Sullivan, B.R., Akin, D.L., Hunt, T., and Cohen, R. (2002, January 11–15). Effects of time delay on telerobotic control of neutral buoyancy vehicles. Proceedings of the 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), Washington, DC, USA.
74. Effect of video lag on laparoscopic surgery: correlation between performance and usability at low latencies10.1002/rcs.1758 ↗
75. Lum, M.J.H., Rosen, J., Lendvay, T.S., Sinanan, M.N., and Hannaford, B. (2009, January 12–17). Effect of time delay on telesurgical performance. Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan.10.1109/ROBOT.2009.5152725 ↗
76. Effects of Visual Display and Motion System Delays on Operator Performance and Uneasiness in a Driving Simulator10.1177/154193128703100502 ↗
77. Ferrier, J.L., Gusikhin, O.Y., Madani, K., and Sasiadek, J.Z. (2013). A System Design for Teleoperated Road Vehicles. ICINCO (2), SciTePress.
78. Hosseini, A., and Lienkamp, M. (2016, January 19–22). Enhancing telepresence during the teleoperation of road vehicles using HMD-based mixed reality. Proceedings of the 2016 IEEE Intelligent Vehicles Symposium (IV), Gotenburg, Sweden.10.1109/IVS.2016.7535568 ↗
79. Monteiro, F., Rocha, P., Menezes, P., Silva, A., and Dias, J. (1997, January 7–11). Teleoperating a mobile robot. A solution based on JAVA language. Proceedings of the ISIE ’97 Proceeding of the IEEE International Symposium on Industrial Electronics, Guimaraes, Portugal.
80. Aykut, T., Zou, C., Xu, J., Van Opdenbosch, D., and Steinbach, E. (2018, January 21–25). A Delay Compensation Approach for Pan-Tilt-Unit-based Stereoscopic 360 Degree Telepresence Systems Using Head Motion Prediction. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8460750 ↗
81. Driving with a Head-Slaved Camera System10.1177/154193120104501812 ↗
82. The Effects of Camera Perspective and Field of View on Performance in Teleoperated Navigation10.1177/154193120605001603 ↗
83. Arthur, K.W. (2000). Effects of Field of View on Performance with Head-Mounted Displays. [Ph.D. Thesis, The University of North Carolina at Chapel Hill].
84. Scribner, D.R., and Gombash, J.W. (1998). The Effect of Stereoscopic and Wide Field of View Conditions on Teleoperator Performance, Army Research Laboratory. Technical Report.
85. Smyth, C., Gombash, J.W., and Burcham, P.M. (2001). Indirect Vision Driving with Fixed Flat Panel Displays for Near-Unity, Wide, and Extended Fields of Camera View, Army Research Laboratory. Technical Report.10.21236/ADA395211 ↗
86. Envisioning human-robot coordination in future operations10.1109/TSMCC.2004.826272 ↗
87. Darken, R.P., Kempster, K., Kempster, M.K., and Peterson, B. (December, January 30). Effects of Streaming Video Quality of Service on Spatial Comprehension in a Reconnaisance Task. Proceedings of the Meeting of I/ITSEC, Orlando, FL, USA.
88. Nonvisually Guided Locomotion to a Previously Viewed Target in Real and Virtual Environments10.1518/001872098779591340 ↗
89. Segregation of form, color, movement, and depth: Anatomy, physiology, and perception10.1126/science.3283936 ↗
90. Theoretical background and conceptual solution for depth perception and eye-hand coordination problems in laparoscopic surgery10.3109/13645709909153166 ↗
91. Depth Perception of Surgeons in Minimally Invasive Surgery10.1177/1553350616639141 ↗
92. Medical telerobotic systems: Current status and future trends10.1186/s12938-016-0217-7 ↗
93. Eye-Hand Coordination during Dynamic Visuomotor Rotations10.1371/journal.pone.0007004 ↗
94. DeJong, B., Colgate, J., and Peshkin, M. (2011). Mental transformations in human-robot interaction. Mixed Reality and Human-Robot Interaction, Springer.10.1007/978-94-007-0582-1_3 ↗
95. The Effect of Automated Compensation for Incongruent Axes on Teleoperator Performance10.1518/001872098779649256 ↗
96. The Role of 2-D and 3-D Task Performance In the Design and Use of Visual Displays10.1177/154193120104500414 ↗
97. Effects of camera arrangement on perceptual-motor performance in minimally invasive surgery10.1037/a0024041 ↗
98. Almeida, L., Menezes, P., and Dias, J. (2017, January 6–8). Improving robot teleoperation experience via immersive interfaces. Proceedings of the 2017 4th Experiment@International Conference (exp.at’17), Faro, Portugal.10.1109/EXPAT.2017.7984414 ↗
99. Working memory10.1016/j.cub.2009.12.014 ↗
100. Maintenance rehearsal and the articulatory loop10.1111/j.2044-8295.1984.tb01921.x ↗