Human-Robot Collaboration: An Augmented Reality Toolkit for Bi-Directional Interaction
Source abstract
Abstract
This work proposes an Augmented Reality (AR) application designed for HoloLens 2 which allows human operators, without particular experience or knowledge of robotics, to easily interact with collaborative robots. Building on the application presented in a previous work of the authors, the novel contributions are focused on a bi-directional interaction that manages the exchange of data from the robot to the human operator and, in the meantime, the flow of commands in the opposite direction. More in detail, the application includes the reading of the robot state, in terms of joint positions, velocities and torques, the visualization of the workspace and the generation and manipulation of the end-effector trajectory by directly moving a set of way-points displayed in the AR environment. Finally, the trajectory feasibility is verified and notified to the user by taking into account the workspace limits. A usability study of the AR platform has been conducted involving 45 participants with different ages and expertise in robot programming and Extended Reality (XR) platforms, comparing two programming methods: a classical kinesthetic teaching interface, provided by the Franka Emika Panda cobot, and the presented AR platform. Participants have reported the effectiveness of the proposed platform, experiencing less physical demand and higher intuitiveness and usability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Augmented Reality Applications · Teleoperation and Haptic Systems · Virtual Reality Applications and Impacts
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Engineering, University of Basilicata, Via dell’Ateneo Lucano 10, 85100 Potenza, Italy
Funding
Italian Ministry of University and Research under the grant PRIN 2022 COM3 (Cooperative Mobile Manipulators for Manufacturing)
References (35)
Show source-provided references
1. Salvato, R., Marra, G., Scardamaglia, P., Di Gironimo, G., Marzullo, D., and Mozzillo, R. (2021, January 9–10). Design and integration of automation systems with manual operation: Small and medium enterprises issues. Proceedings of the Second International Conference on Design Tools and Methods in Industrial Engineering, ADM 2021, Rome, Italy.10.1007/978-3-030-91234-5_30 ↗
2. Determinant assembly approach for flat-shaped airframe components10.1007/s00170-020-05459-8 ↗
3. Sheridan, T.B. (2002). Humans and Automation: System Design and Research Issues, John Wiley and Sons.
4. Cobots in industry 4.0: A roadmap for future practice studies on human–robot collaboration10.1109/THMS.2021.3092684 ↗
5. De Franco, A., Lamon, E., Balatti, P., De Momi, E., and Ajoudani, A. (2019, January 3–5). An Intuitive augmented reality interface for task scheduling, monitoring, and work performance improvement in human-robot collaboration. Proceedings of the 2019 IEEE International Work Conference on Bioinspired Intelligence (IWOBI), Budapest, Hungary.10.1109/IWOBI47054.2019.9114472 ↗
6. Human-robot collaboration: A literature review and augmented reality approach in design10.5772/5664 ↗
7. (2023, July 05). HoloLens 2. Available online: https://www.microsoft.com/en-us/hololens/hardware.
8. Calzone, N., Sileo, M., Mozzillo, R., Pierri, F., and Caccavale, F. (2022, January 1–3). Mixed Reality Platform Supporting Human-Robot Interaction. Proceedings of the International Joint Conference on Mechanics, Design Engineering & Advanced Manufacturing, JCM 2022, Ischia, Italy.10.1007/978-3-031-15928-2_102 ↗
9. (2023, July 11). Franka Emika Panda. Available online: https://www.franka.de/.
10. (2023, June 29). Unity. Available online: https://unity.com/.
11. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009, January 12–17). ROS: An open-source Robot Operating System. Proceedings of the ICRA Workshop on Open Source Software, Kobe, Japan.10.1109/MRA.2010.936956 ↗
12. GitHub (2023, July 06). ROS#. Available online: https://github.com/siemens/ros-sharp.
13. Suzuki, R., Karim, A., Xia, T., Hedayati, H., and Marquardt, N. (May, January 29). Augmented Reality and Robotics: A Survey and Taxonomy for AR-Enhanced Human-Robot Interaction and Robotic Interfaces. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA.10.1145/3491102.3517719 ↗
14. SUS: A quick and dirty usability scale
15. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research10.1016/S0166-4115(08)62386-9 ↗
16. Romero, D., Stahre, J., Wuest, T., Noran, O., Bernus, P., Fast-Berglund, Å., and Gorecky, D. (2016, January 29–31). Towards an operator 4.0 typology: A human-centric perspective on the fourth industrial revolution technologies. Proceedings of the International Conference on Computers and Industrial Engineering (CIE46), Tianjin, China.
17. Human-machine interaction sensing technology based on hand gesture recognition: A review10.1109/THMS.2021.3086003 ↗
18. A new mixed-reality-based teleoperation system for telepresence and maneuverability enhancement10.1109/THMS.2019.2960676 ↗
19. (2023, July 05). Microsoft Mixed Reality Toolkit. Available online: https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/?view=mrtkunity-2021-05.
20. Communicating and controlling robot arm motion intent through mixed-reality head-mounted displays10.1177/0278364919842925 ↗
21. Gadre, S.Y., Rosen, E., Chien, G., Phillips, E., Tellex, S., and Konidaris, G. (2019, January 20–24). End-User Robot Programming Using Mixed Reality. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793988 ↗
22. Quintero, C.P., Li, S., Pan, M.K., Chan, W.P., Machiel Van der Loos, H., and Croft, E. (2018, January 1–5). Robot Programming Through Augmented Trajectories in Augmented Reality. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593700 ↗
23. Kästner, L., and Lambrecht, J. (2019, January 18–20). Augmented-reality-based visualization of navigation data of mobile robots on the microsoft hololens-possibilities and limitations. Proceedings of the 2019 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), Bangkok, Thailand.10.1109/CIS-RAM47153.2019.9095836 ↗
24. Ostanin, M., Mikhel, S., Evlampiev, A., Skvortsova, V., and Klimchik, A. (August, January 31). Human-robot interaction for robotic manipulator programming in Mixed Reality. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9196965 ↗
25. (2023, June 30). MoveIt. Available online: https://moveit.ros.org/.
26. GitHub (2023, June 28). ROS# UWP. Available online: https://github.com/EricVoll/ros-sharp.
27. (2023, July 11). Andrej Orsula, Panda_ign. Available online: https://github.com/AndrejOrsula/panda_ign.
28. ROS Wiki (2023, June 28). URDF. Available online: http://wiki.ros.org/urdf.
29. Microsoft (2023, July 05). QR Tracking Unity Sample for HoloLens 2 Using OpenXR. Available online: https://github.com/yl-msft/QRTracking.
30. GitHub (2023, July 12). NuGet. Available online: https://github.com/GlitchEnzo/NuGetForUnity.
31. Siciliano, B., Sciavicco, L., Villani, L., and Oriolo, G. (2009). Robotics–Modelling, Planning and Control, Springer.10.1007/978-1-84628-642-1 ↗
32. (2023, July 12). JointState Message. Available online: http://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/JointState.html.
33. Nasa-Task Load Index (NASA-TLX); 20 Years Later10.1177/154193120605000909 ↗
34. Analyzing and interpreting data from Likert-type scales10.4300/JGME-5-4-18 ↗
35. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, Lawrence Erlbaum Associates, Publishers. [2nd ed.].