Fusing Hand Postures and Speech Recognition for Tasks Performed by an Integrated Leg–Arm Hexapod Robot
Source abstract
Abstract
Hand postures and speech are convenient means of communication for humans and can be used in human–robot interaction. Based on structural and functional characteristics of our integrated leg-arm hexapod robot, to perform reconnaissance and rescue tasks in public security application, a method of linkage of movement and manipulation of robots is proposed based on the visual and auditory channels, and a system based on hand postures and speech recognition is described. The developed system contains: a speech module, hand posture module, fusion module, mechanical structure module, control module, path planning module and a 3D SLAM (Simultaneous Localization and Mapping) module. In this system, three modes, i.e., the hand posture mode, speech mode, and a combination of the hand posture and speech modes, are used in different situations. The hand posture mode is used for reconnaissance tasks, and the speech mode is used to query the path and control the movement and manipulation of the robot. The combination of the two modes can be used to avoid ambiguity during interaction. A semantic understanding-based task slot structure is developed by using the visual and auditory channels. In addition, a method of task planning based on answer-set programming is developed, and a system of network-based data interaction is designed to control movements of the robot using Chinese instructions remotely based on a wide area network. Experiments were carried out to verify the performance of the proposed system.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Robot Manipulation and Learning · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Robotics Institute, School of Mechanical Engineering and Automation, Beihang University, Beijing 100191, China
Beijing Institute of Computer and Electronics Application, Beijing 100191, China
First Research Institute of the Ministry of Public Security of People’s Republic of China, Beijing 100191, China
Funding
Natural Science Foundation of Beijing Municipality · National Natural Science Foundation of China
References (20)
Show source-provided references
1. Visual interpretation of hand gestures for human-computer interaction: A review10.1109/34.598226 ↗
2. Hand posture and gesture recognition technology10.1007/s00521-014-1574-4 ↗
3. New approach for static gesture recognition
4. Stiefelhagen, R., Fogen, C., Gieselmann, P., Holzapfel, H., Nickel, K., and Waibel, A. (October, January 28). Natural human-robot interaction using speech, headNatural human-robot interaction using speech, head pose and gestures. Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sendai, Japan.
5. Enabling multimodal human–robot interaction for the karlsruhe humanoid robot10.1109/TRO.2007.907484 ↗
6. Visual recognition of pointing gestures for human–robot interaction10.1016/j.imavis.2005.12.020 ↗
7. Real-time person tracking and pointing gesture recognition for human-robot interaction10.1007/978-3-540-24837-8_4 ↗
8. Nickel, K., and Stiefelhagen, R. (2003, January 5–7). Pointing gesture recognition based on 3D-tracking of face, hands and head orientation. Proceedings of the 5th International Conference on Multimodal Interfaces, Vancouver, BC, Canada.10.1145/958432.958460 ↗
9. Seemann, E., Nickel, K., and Stiefelhagen, R. (2004, January 19). Head pose estimation using stereo vision for human-robot interaction. Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition, Seoul, Korea.
10. Two-handed gesture recognition and fusion with speech to command a robot10.1007/s10514-011-9263-y ↗
11. Deep learning-based multimodal control interface for human-robot collaboration10.1016/j.procir.2018.03.224 ↗
12. Multi-modal interaction of human and home robot in the context of room map generation10.1023/A:1019689509522 ↗
13. Rogalla, O., Ehrenmann, M., Zollner, R., Becher, R., and Dillmann, R. (2002, January 27). Using gesture and speech control for commanding a robot assistant. Proceedings of the 11th IEEE International Workshop on Robot and Human Interactive Communication, Berlin, Germany.
14. Pick-and-place application development using voice and visual commands10.1108/01439911211268796 ↗
15. Van Delden, S., and Umrysh, M.A. (2011, January 17–18). Visual detection of objects in a robotic work area using hand gestures. Proceedings of the 2011 IEEE International Symposium on Robotic and Sensors Environments (ROSE), Montreal, QC, Canada.10.1109/ROSE.2011.6058529 ↗
16. Chen, X., and Qin, S. (2017, January 20–22). Approach to high efficient hierarchical pathfinding of indoor mobile service robots based on grid map and Floyd-Warshall algorithm. Proceedings of the 2017 Chinese Automation Congress (CAC), Jinan, China.10.1109/CAC.2017.8243944 ↗
17. Chen, X., and Qin, S. (2017, January 20–22). An effective approach to SLAM toward autonomous operation for a Leg/Arm composite mobile robot in unknown environment based on RGB-D images. Proceedings of the 2017 Chinese Automation Congress (CAC), Jinan, China.
18. Li, W., Xu, K., Qi, J., and Ding, X. (2018, January 12–15). A Natural Language Processing Method of Chinese Instruction for Multi-legged Manipulating Robot. Proceedings of the 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia.10.1109/ROBIO.2018.8664888 ↗
19. Law, H., Teng, Y., Russakovsky, O., and Deng, J. (2019). CornerNet-Lite: Efficient Keypoint Based Object Detection. arXiv.
20. Lafferty, J., McCallum, A., and Pereira, F.C. (July, January 28). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA.