Cooperative and Multimodal Capabilities Enhancement in the CERNTAURO Human–Robot Interface for Hazardous and Underwater Scenarios
Source abstract
Abstract
The use of remote robotic systems for inspection and maintenance in hazardous environments is a priority for all tasks potentially dangerous for humans. However, currently available robotic systems lack that level of usability which would allow inexperienced operators to accomplish complex tasks. Moreover, the task’s complexity increases drastically when a single operator is required to control multiple remote agents (for example, when picking up and transporting big objects). In this paper, a system allowing an operator to prepare and configure cooperative behaviours for multiple remote agents is presented. The system is part of a human–robot interface that was designed at CERN, the European Center for Nuclear Research, to perform remote interventions in its particle accelerator complex, as part of the CERNTAURO project. In this paper, the modalities of interaction with the remote robots are presented in detail. The multimodal user interface enables the user to activate assisted cooperative behaviours according to a mission plan. The multi-robot interface has been validated at CERN in its Large Hadron Collider (LHC) mockup using a team of two mobile robotic platforms, each one equipped with a robotic manipulator. Moreover, great similarities were identified between the CERNTAURO and the TWINBOT projects, which aim to create usable robotic systems for underwater manipulations. Therefore, the cooperative behaviours were validated within a multi-robot pipe transport scenario in a simulated underwater environment, experimenting more advanced vision techniques. The cooperative teleoperation can be coupled with additional assisted tools such as vision-based tracking and grasping determination of metallic objects, and communication protocols design. The results show that the cooperative behaviours enable a single user to face a robotic intervention with more than one robot in a safer way.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Sensor-Based Localization · Underwater Vehicles and Communication Systems · Modular Robots and Swarm Intelligence
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
European Organization for Nuclear Research, Esplanade des Particules 1, 1217 Meyrin, Switzerland
Interactive Robotic Systems Lab, Jaume I University of Castellón, 12006 Castellón de la Plana, Spain
References (44)
Show source-provided references
1. CERNTAURO: A Modular Architecture for Robotic Inspection and Telemanipulation in Harsh and Semi-Structured Environments10.1109/ACCESS.2018.2849572 ↗
2. Multimodal Human-Robot Interface for Accessible Remote Robotic Interventions in Hazardous Environments10.1109/ACCESS.2019.2939493 ↗
3. Di Castro, M., Masi, A., Buonocore, L.R., Ferre, M., Losito, R., Gilardoni, S., and Lunghi, G. (2017, January 8–13). A dual arms robotic platform control for navigation, inspection and telemanipulation. Proceedings of the 16th International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS 2017), Barcelona, Spain.
4. Handbook of Accelerator Physics and Engineering, 2nd edition, by Alexander Wu Chao, Karl Hubert Mes, Maury Tigner and Frank Zimmermann10.1080/00107514.2014.885574 ↗
5. Real-time rate distortion-optimized image compression with region of interest on the ARM architecture for underwater robotics applications10.1007/s11554-018-0833-5 ↗
6. Centelles, D., Soriano, A., Martín, J.V., Marín, R., and Sanz, P.J. (2019). Underwater Wireless Communications for Cooperative Robotics with UWSim-NET. Appl. Sci., 9.10.3390/app9173526 ↗
7. I-AUV Mechatronics Integration for the TRIDENT FP7 Project10.1109/TMECH.2015.2395413 ↗
8. Centelles, D., Moscoso, E., Vallicrosa, G., Palomeras, N., Sales, J., Marti, J.V., Marin, R., Ridao, P., and Sanz, P.J. (2017, January 19–22). Wireless HROV control with compressed visual feedback over an acoustic link. Proceedings of the OCEANS 2017, Aberdeen, UK.10.1109/OCEANSE.2017.8084979 ↗
9. Human-robot interaction in rescue robotics10.1109/TSMCC.2004.826267 ↗
10. Chen, J.Y., Durlach, P.J., Sloan, J.A., and Bowens, L.D. (2005). Robotic Operator Performance in Simulated Reconnaissance Missions, Army Research Lab Aberdeen Proving Ground Md Human Research and Engineering Directorate. Technical Report.
11. Impacting system design with human performance modeling and experiment: Another success story10.1177/154193120605002303 ↗
12. Mitchell, D., and Henthorn, T. (2005). Soldier Workload Analysis of the Mounted Combat System (MCS) Platoon’s Use of Unmanned Assets, Army Research Laboratory, Aberdeen Proving Ground. Tech Report, ARL-TR-3476.
13. Wickens, C.D., Hollands, J.G., Banbury, S., and Parasuraman, R. (2015). Engineering Psychology and Human Performance, Psychology Press.10.4324/9781315665177 ↗
14. Sharma, M., Young, J.E., and Eskicioglu, R. (2012). Developing guidelines for in-the-field control of a team of robots. Proceedings of the Seventh Annual ACM/IEEE International conference on Human-Robot Interaction-HRI ’12, ACM Press.10.1145/2157689.2157771 ↗
15. Zgelen, A.T., and Sklar, E.I. (2014). Modeling and Analysis of Task Complexity in Single-Operator Multi-Robot Teams. Proceedings of the 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI), ACM.10.1145/2559636.2559843 ↗
16. Bradshaw, J.M., Dignum, V., Jonker, C.M., and Sierhuis, M. (2012). Human-agent-robot teamwork. Proceedings of the 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI), ACM.10.1145/2157689.2157843 ↗
17. Sheridan, T.B., and Verplank, W.L. (1978). Human and Computer Control of Undersea Teleoperators, Massachusetts Inst of Tech Cambridge Man-Machine Systems Lab. Technical Report.10.21236/ADA057655 ↗
18. Miller, C.A., Funk, H.B., Dorneich, M., and Whitlow, S.D. (2002, January 27–31). A playbook interface for mixed initiative control of multiple unmanned vehicle teams. Proceedings of the 21st Digital Avionics Systems Conference, Irvine, CA, USA.
19. Mitchell, P., Cummings, M., and Sheridan, T. (2005, January 13–16). Management of multiple dynamic human supervisory control tasks. Proceedings of the 10th International Command and Control Research and Technology Symposium, McLean, VA, USA.
20. Goodrich, M.A., Quigley, M., and Cosenzo, K. (2005). Task switching and multi-robot teams. Multi-Robot Systems. From Swarms to Intelligent Automata Volume III, Springer.10.1007/1-4020-3389-3_15 ↗
21. Lomas, M., Moffitt, V.Z., Craven, P., Cross, E.V., Franke, J.L., and Taylor, J.S. (2011). Team-based interactions with heterogeneous robots through a novel HRI software architecture. Proceedings of the 6th International Conference on Human-Robot interaction-HRI ’11, ACM Press.10.1145/1957656.1957727 ↗
22. Bagosi, T., Hindriks, K.V., and Neerincx, M.A. (2016, January 7–10). Ontological reasoning for human-robot teaming in search and rescue missions. Proceedings of the 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Christchurch, New Zealand.10.1109/HRI.2016.7451873 ↗
23. Scholtz, J., and Bahrami, S. (2003, January 8). Human-robot interaction: Development of an evaluation methodology for the bystander role of interaction. Proceedings of the 2003 IEEE International Conference on Systems, Man and Cybernetics, Washington, DC, USA.
24. (2019). Ergonomics of Human-System Interaction—Part 210: Human-Centred Design for Interactive Systems, Korean Standards Association.
25. Prats, M., Fernández, J.J., and Sanz, P.J. (2012, January 7–12). Combining Template Tracking and Laser Peak Detection for 3D Reconstruction and Grasping in Underwater Environments. Proceedings of the International Conference on Intelligent Robots and Systems, Vilamoura, Portugal.10.1109/IROS.2012.6385792 ↗
26. Visually-guided manipulation techniques for robotic autonomous underwater panel interventions10.1016/j.arcontrol.2015.09.012 ↗
27. Zhang, H., Lan, X., Zhou, X., Wang, J., and Zheng, N. (August, January 31). Vision-based Robotic Grasp Success Determination with Convolutional Neural Network. Proceedings of the 2017 IEEE 7th Annual International Conference on CYBER Technology in Automation Control, and Intelligent Systems (CYBER), Waikiki Beach, HI, USA.10.1109/CYBER.2017.8446360 ↗
28. Robust 3D object tracking from monocular images using stable parts10.1109/TPAMI.2017.2708711 ↗
29. Hauck, A., Ruttinger, J., Sorg, M., and Farber, G. (1999, January 17–21). Visual determination of 3D grasping points on unknown objects with a binocular camera system. Proceedings of the 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No.99CH36289), Kyongju, Korea.
30. Veiga Almagro, C., Di Castro, M., Lunghi, G., Marín Prades, R., Sanz Valero, P.J., Pérez, M.F., and Masi, A. (2019). Monocular Robust Depth Estimation Vision System for Robotic Tasks Interventions in Metallic Targets. Sensors, 19.10.3390/s19143220 ↗
31. Multi-task cooperative control in a heterogeneous ground-air robot team10.1016/j.ifacol.2015.06.463 ↗
32. Sasaki, J., Maeda, Y., Aiyama, Y., Ota, J., and Arai, T. (1996, January 22–28). Initial grasping strategy for an unknown object by cooperative mobile robots. Proceedings of the IEEE International Conference on Robotics and Automation, Minneapolis, MN, USA.
33. Liu, Z., Kamogawa, H., and Ota, J. (2010, January 21–22). Manipulation of an irregularly shaped object by two mobile robots. Proceedings of the 2010 IEEE/SICE International Symposium on System Integration, Sendai, Japan.10.1109/SII.2010.5708328 ↗
34. Linking Acoustic Communications and Network Performance: Integration and Experimentation of an Underwater Acoustic Network10.1109/JOE.2013.2279472 ↗
35. Di Castro, M., Lunghi, G., Masi, A., Ferre, M., and Prades, R.M. (2019, January 25–27). A Multidimensional RSSI Based Framework for Autonomous Relay Robots in Harsh Environments. Proceedings of the 2019 Third IEEE International Conference on Robotic Computing (IRC), Naples, Italy.10.1109/IRC.2019.00035 ↗
36. Re-Evaluation of RF Electromagnetic Communication in Underwater Sensor Networks10.1109/MCOM.2010.5673085 ↗
37. Dea, J., Radosevic, D., Tran, N., Chavez, J., and Neuner, B. (2017). Land and Undersea Field Testing of Very Low Frequency RF Antennas and Loop Transceivers, Spawar Systems Center (SSC) Pacific. Technical Report.
38. Underwater Optical Wireless Communication10.1109/ACCESS.2016.2552538 ↗
39. Rubino, E.M., Centelles, D., Sales, J., Marti, J.V., Marin, R., Sanz, P.J., and Alvares, A.J. (2017, January 19–22). Progressive image compression and transmission with region of interest in underwater robotics. Proceedings of the OCEANS 2017-Aberdeen, Aberdeen, UK.10.1109/OCEANSE.2017.8084999 ↗
40. Lefevre, C. (2019, February 01). The CERN Accelerator Complex (REPORT CERN-DI-0812015). Available online: http://cds.cern.ch/record/1260465.
41. Speeded-up robust features (SURF)10.1016/j.cviu.2007.09.014 ↗
42. Di Castro, M., Almagro, C.V., Lunghi, G., Marin, R., Ferre, M., and Masi, A. (2018, January 1–5). Tracking-Based Depth Estimation of Metallic Pieces for Robotic Guidance. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8594229 ↗
43. Hou, X., and Zhang, L. (2007, January 18–23). Saliency detection: A spectral residual approach. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.10.1109/CVPR.2007.383267 ↗
44. Sanz, P., Requena, A., Inesta, J., and Pobil, A. (1994). Grasping the Not-So-Obvious: Vision-Based Object Handling for Industrial Applications, Institute of Electrical and Electronics Engineers.