The Teleoperation of Robot Arms by Interacting with an Object’s Digital Twin in a Mixed Reality Environment
Source abstract
Abstract
The teleoperation of robot arms can prevent users from working in hazardous environments, but current teleoperation uses a 2D display and controls the end effector of robot arms, which introduces the problem of a limited view and complex operations. In this study, a teleoperation method for robot arms is proposed, which can control the robot arm by interacting with the digital twins of objects. Based on the objects in the workspace, this method generates a virtual scene containing digital twins. Users can observe the virtual scene from any direction and move the digital twins of the objects at will to control the robot arm. This study compared the proposed method and the traditional method, which uses a 2D display and a game controller, through a pick-and-place task. The proposed method achieved 45% lower scores in NASA-TLX and 31% higher scores in SUS than traditional teleoperation methods. The results indicate that the proposed method can reduce the workload and improve the usability of teleoperation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Digital Transformation in Industry · Robotics and Automated Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130022, China
Jilin Provincial International Joint Research Center of Brain Informatics and Intelligence Science, Changchun 130022, China
Zhongshan Institute of Changchun University of Science and Technology, Zhongshan 528437, China
Funding
Education Department of Jilin Province
References (36)
Show source-provided references
1. A LoRa-Based Disaster Management System for Search and Rescue Mission10.1109/JIOT.2024.3435734 ↗
2. A novel pose estimation method for robot threaded assembly pre-alignment based on binocular vision10.1016/j.rcim.2024.102939 ↗
3. Human–robot vs. human–manual teams: Understanding the dynamics of experience and performance variability in picker-to-parts order picking10.1016/j.cie.2024.110750 ↗
4. Wang, F., Li, C., Niu, S., Wang, P., Wu, H., and Li, B. (2022). Design and Analysis of a Spherical Robot with Rolling and Jumping Modes for Deep Space Exploration. Machines, 10.10.3390/machines10020126 ↗
5. Yuan, S., Chen, R., Zang, L., Wang, A., Fan, N., Du, P., Xi, Y., and Wang, T. (2024). Development of a software system for surgical robots based on multimodal image fusion: Study protocol. Front. Surg., 11.10.3389/fsurg.2024.1389244 ↗
6. Cooper, R.A., Smolinski, G., Candiotti, J.L., Satpute, S., Grindle, G.G., Sparling, T.L., Nordstrom, M.J., Yuan, X., Symsack, A., and Dae Lee, C. (2024). Current State, Needs, and Opportunities for Wearable Robots in Military Medical Rehabilitation and Force Protection. Actuators, 13.10.3390/act13070236 ↗
7. Residual Reinforcement Learning for Motion Control of a Bionic Exploration Robot—RoboDact
8. The construction industry as a loosely coupled system: Implications for productivity and innovation10.1080/01446190210163543 ↗
9. Construction automation and robotics for high-rise buildings over the past decades: A comprehensive review10.1016/j.aei.2019.100989 ↗
10. Su, Y.-P., Chen, X.-Q., Zhou, C., Pearson, L.H., Pretty, C.G., and Chase, J.G. (2023). Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions. Appl. Sci., 13.10.3390/app132212129 ↗
11. Force reflecting joystick control for applications to bilateral teleoperation in construction machinery10.1007/s12541-017-0038-z ↗
12. Sensorless force feedback joystick control for teleoperation of construction equipment10.1007/s12541-017-0113-5 ↗
13. A New 6-DOF Haptic Device for Teleoperation of 6-DOF Serial Robots10.1109/TIM.2011.2164285 ↗
14. Comparative Analysis of 3-D Robot Teleoperation Interfaces With Novice Users10.1109/TSMCB.2009.2038357 ↗
15. Teleoperation of industrial robot manipulators based on augmented reality10.1007/s00170-020-05997-1 ↗
16. Caterino, M., Rinaldi, M., Di Pasquale, V., Greco, A., Miranda, S., and Macchiaroli, R. (2023). A Human Error Analysis in Human–Robot Interaction Contexts: Evidence from an Empirical Study. Machines, 11.10.3390/machines11070670 ↗
17. Video see-through augmented reality fire safety training: A comparison with virtual reality and video training10.1016/j.ssci.2024.106714 ↗
18. Madani, S., Sayadi, A., Turcotte, R., Cecere, R., Aoude, A., and Hooshiar, A. (2025). A universal calibration framework for mixed-reality assisted surgery. Comput. Methods Programs Biomed., 259.10.1016/j.cmpb.2024.108470 ↗
19. Integrative human and object aware online progress observation for human-centric augmented reality assembly10.1016/j.aei.2024.103081 ↗
20. An Effective Head-Based HRI for 6D Robotic Grasping Using Mixed Reality10.1109/LRA.2023.3261701 ↗
21. Gadre, S.Y., Rosen, E., Chien, G., Phillips, E., Tellex, S., and Konidaris, G. (2019, January 20–24). End-User Robot Programming Using Mixed Reality. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793988 ↗
22. Toward Programming a Collaborative Robot by Interacting with Its Digital Twin in a Mixed Reality Environment
23. Cruz Ulloa, C., Domínguez, D., Del Cerro, J., and Barrientos, A. (2022). A Mixed-Reality Tele-Operation Method for High-Level Control of a Legged-Manipulator Robot. Sensors, 22.10.3390/s22218146 ↗
24. DePauw, C.G., Univerisity, B.R., Lehigh, A.W., Miller, M., and Stoytchev, A. (2008, January 3–6). An Effective and Intuitive Control Interface for Remote Robot Teleoperation with Complete Haptic Feedback. Proceedings of the Emerging Technologies Conference-ETC, San Diego, CA, USA.
25. Haptic and Visual Feedback Assistance for Dual-Arm Robot Teleoperation in Surface Conditioning Tasks10.1109/TOH.2020.3004388 ↗
26. Chen, F., Gao, B., Selvaggio, M., Li, Z., Caldwell, D., Kershaw, K., Masi, A., Castro, M.D., and Losito, R. (2016, January 7–10). A framework of teleoperated and stereo vision guided mobile manipulation for industrial automation. Proceedings of the 2016 IEEE International Conference on Mechatronics and Automation, Harbin, China.10.1109/ICMA.2016.7558810 ↗
27. McHenry, N., Spencer, J., Zhong, P., Cox, J., Amiscaray, M., Wong, K.C., and Chamitoff, G. (2021, January 6–13). Predictive XR Telepresence for Robotic Operations in Space. Proceedings of the 2021 IEEE Aerospace Conference (50100), Big Sky, MT, USA.10.1109/AERO50100.2021.9438161 ↗
28. Smolyanskiy, N., and González-Franco, M. (2017). Stereoscopic First Person View System for Drone Navigation. Front. Robot. AI, 4.10.3389/frobt.2017.00011 ↗
29. Communicating and controlling robot arm motion intent through mixed-reality head-mounted displays10.1177/0278364919842925 ↗
30. Mixed reality-integrated 3D/2D vision mapping for intuitive teleoperation of mobile manipulator10.1016/j.rcim.2022.102332 ↗
31. Pace, F.D., Gorjup, G., Bai, H., Sanna, A., Liarokapis, M., and Billinghurst, M. (2020, January 1–4). Assessing the Suitability and Effectiveness of Mixed Reality Interfaces for Accurate Robot Teleoperation. Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology, Virtual Event, Canada.
32. Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction10.1016/j.aei.2020.101170 ↗
33. A New Mixed-Reality-Based Teleoperation System for Telepresence and Maneuverability Enhancement10.1109/THMS.2019.2960676 ↗
34. Dual-arm robot teleoperation support with the virtual world10.1007/s10015-020-00587-6 ↗
35. The Vicarios Virtual Reality Interface for Remote Robotic Teleoperation10.1007/s10846-021-01311-7 ↗
36. Sun, Q., Chen, W., Chao, J., Lin, W., Xu, Z., and Cao, R. (2023). Smart Task Assistance in Mixed Reality for Astronauts. Sensors, 23.10.3390/s23094344 ↗