Deep Learning for Intelligent Human–Computer Interaction
Source abstract
Abstract
In recent years, gesture recognition and speech recognition, as important input methods in Human–Computer Interaction (HCI), have been widely used in the field of virtual reality. In particular, with the rapid development of deep learning, artificial intelligence, and other computer technologies, gesture recognition and speech recognition have achieved breakthrough research progress. The search platform used in this work is mainly the Google Academic and literature database Web of Science. According to the keywords related to HCI and deep learning, such as “intelligent HCI”, “speech recognition”, “gesture recognition”, and “natural language processing”, nearly 1000 studies were selected. Then, nearly 500 studies of research methods were selected and 100 studies were finally selected as the research content of this work after five years (2019–2022) of year screening. First, the current situation of the HCI intelligent system is analyzed, the realization of gesture interaction and voice interaction in HCI is summarized, and the advantages brought by deep learning are selected for research. Then, the core concepts of gesture interaction are introduced and the progress of gesture recognition and speech recognition interaction is analyzed. Furthermore, the representative applications of gesture recognition and speech recognition interaction are described. Finally, the current HCI in the direction of natural language processing is investigated. The results show that the combination of intelligent HCI and deep learning is deeply applied in gesture recognition, speech recognition, emotion recognition, and intelligent robot direction. A wide variety of recognition methods were proposed in related research fields and verified by experiments. Compared with interactive methods without deep learning, high recognition accuracy was achieved. In Human–Machine Interfaces (HMIs) with voice support, context plays an important role in improving user interfaces. Whether it is voice search, mobile communication, or children’s speech recognition, HCI combined with deep learning can maintain better robustness. The combination of convolutional neural networks and long short-term memory networks can greatly improve the accuracy and precision of action recognition. Therefore, in the future, the application field of HCI will involve more industries and greater prospects are expected.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Educational Technology and Pedagogy · Advanced Technologies in Various Fields
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Game Design, Faculty of Arts, Uppsala University, SE-62167 Uppsala, Sweden
Technologies of Vision, Digital Industry Center, Fondazione Bruno Kessler, Via Sommarive 18, 38123 Trento, Italy
Amazon AWS AI, Seattle, WA 98125, USA
Instituto de Investigacion para la Gestion Integrada de Zonas Costeras, Universitat Politecnica de Valencia, 46022 Valencia, Spain
Security and Optimization for Networked Globe Laboratory (SONG Lab), Embry-Riddle Aeronautical University, Daytona Beach, FL 32114, USA
References (100+)
Show source-provided references
1. Multi-Platform Intelligent System for Multimodal Human-Computer Interaction10.31577/cai_2021_1_83 ↗
2. Content based video retrieval system based on multimodal feature grouping by KFCM clustering algorithm to promote human–computer interaction10.1007/s12652-020-02190-w ↗
3. Emotion Recognition Using WT-SVM in Human-Computer Interaction10.32604/jnm.2020.010674 ↗
4. Research on Application of Cognitive-Driven Human-Computer Interaction
5. An audiovisual interface-based drumming system for multimodal human–robot interaction10.1007/s12193-020-00352-w ↗
6. Raptis, G., Kavvetsos, G., and Katsini, C. (2021). MuMIA: Multimodal Interactions to Better Understand Art Contexts. Appl. Sci., 11.10.3390/app11062695 ↗
7. Wang, J., Cheng, R., Liu, M., and Liao, P.-C. (2021). Research Trends of Human–Computer Interaction Studies in Construction Hazard Recognition: A Bibliometric Review. Sensors, 21.10.3390/s21186172 ↗
8. Multimodal Fused Emotion Recognition About Expression-EEG Interaction and Collaboration Using Deep Learning10.1109/ACCESS.2020.3010311 ↗
9. Different Contextual Window Sizes Based RNNs for Multimodal Emotion Detection in Interactive Conversations10.1109/ACCESS.2020.3005664 ↗
10. A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions10.1016/j.knosys.2021.106970 ↗
11. Human-computer multimodal interface to internet navigation10.1080/17483107.2020.1799440 ↗
12. A Human–Computer Interaction framework for emotion recognition through time-series thermal video sequences10.1016/j.compeleceng.2021.107280 ↗
13. Depth Sequential Information Entropy Maps and Multi-Label Subspace Learning for Human Action Recognition10.1109/ACCESS.2020.3006067 ↗
14. Development of computer-based photoshop learning media using computer based interaction method
15. Liu, X., and Zhang, L. (2021). Design and Implementation of Human-Computer Interaction Adjustment in Nuclear Power Monitoring System. Mi-croprocessors and Microsystems. Microprocess. Microsyst., 104096.10.1016/j.micpro.2021.104096 ↗
16. Research on Multimodal Perceptual Navigational Virtual and Real Fusion Intelligent Experiment Equipment and Algorithm10.1109/ACCESS.2020.2978089 ↗
17. Demonstrating the feasibility of multimodal neuroimaging data capture with a wearable electoencephalography + functional near-infrared spectroscopy (eeg+fnirs) in situ10.1017/pds.2021.90 ↗
18. Research on Human-Computer Interaction Control Method in the Background of Internet of Things10.1142/S0219265921430155 ↗
19. Relationship Development with Humanoid Social Robots: Applying Interpersonal Theories to Human–Robot Interaction10.1089/cyber.2020.0181 ↗
20. Social cognition in the age of human–robot interaction10.1016/j.tins.2020.03.013 ↗
21. Robots in groups and teams: A literature review10.1145/3415247 ↗
22. Should I Blame the Human or the Robot? Attribution within a Human–Robot Group10.1007/s12369-020-00645-w ↗
23. Iio, T., Yoshikawa, Y., Chiba, M., Asami, T., Isoda, Y., and Ishiguro, H. (2020). Twin-Robot Dialogue System with Robustness against Speech Recognition Failure in Human-Robot Dialogue with Elderly People. Appl. Sci., 10.10.3390/app10041522 ↗
24. Pan, S. (2021). Design of intelligent robot control system based on human–computer interaction. Int. J. Syst. Assur. Eng. Manag., 1–10.10.1007/s13198-021-01267-9 ↗
25. An optimal Electrical Impedance Tomography drive pattern for human-computer interaction applications
26. Human–robot interaction based on gesture and movement recognition10.1016/j.image.2019.115686 ↗
27. ICIS 2019 SIGHCI workshop panel report: Human–computer interaction challenges and opportunities for fair, trustworthy and ethical artificial intelligence10.17705/1thci.00130 ↗
28. Interactive design of intelligent machine vision based on human–computer interaction mode10.1016/j.micpro.2020.103059 ↗
29. Destruction, Catharsis, and Emotional Release in Human-Robot Interaction10.1145/3385007 ↗
30. Understanding human-robot teams in light of all-human teams: Aspects of team interaction and shared cognition10.1016/j.ijhcs.2020.102436 ↗
31. Research Trends in Social Robots for Learning10.1007/s43154-020-00008-3 ↗
32. Human-Computer Interaction based Joint Attention cues: Implications on functional and physiological measures for children with autism spectrum disorder10.1016/j.chb.2019.106163 ↗
33. Home-care Professionals’ Ethical Perceptions of the Development and Use of Home-care Robots for Older Adults in Japan10.1080/10447318.2020.1736809 ↗
34. A conceptual framework to evaluate human-robot collaboration10.1007/s00170-020-05363-1 ↗
35. Pretto, N., and Poiesi, F. (2017, January 28–29). Towards gesture-based multi-user interactions in collaborative virtual environments. Proceedings of the 5th International Workshop LowCost 3D-Sensors, Algorithms, Applications, Hamburg, Germany.
36. Pani, M., and Poiesi, F. (2018). Distributed data exchange with Leap Motion. International Conference on Augmented Reality, Virtual Reality and Computer Graphics, Springer.10.1007/978-3-319-95282-6_46 ↗
37. Ensemble deep learning in bioinformatics10.1038/s42256-020-0217-y ↗
38. Deep learning for tomographic image reconstruction10.1038/s42256-020-00273-z ↗
39. Deep-Learning-Empowered Breast Cancer Auxiliary Diagnosis for 5GB Remote E-Health10.1109/MWC.001.2000374 ↗
40. Application of deep learning for fast detection of COVID-19 in X-rays using nCOVnet10.1016/j.chaos.2020.109944 ↗
41. Deep learning for the design of photonic structures10.1038/s41566-020-0685-y ↗
42. A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis10.1183/13993003.00775-2020 ↗
43. Deep Learning–based Text Classification: A Comprehensive Review10.1145/3439726 ↗
44. Deep learning in environmental remote sensing: Achievements and challenges10.1016/j.rse.2020.111716 ↗
45. Evaluation of an Automatic Speech Recognition Platform for Dysarthric Speech10.1159/000511042 ↗
46. End-to-End Audiovisual Speech Recognition System with Multitask Learning10.1109/TMM.2020.2975922 ↗
47. Bhatt, S., Jain, A., and Dev, A. (2021). Continuous Speech Recognition Technologies—A Review. Recent Developments in Acoustics, Springer.10.1007/978-981-15-5776-7_8 ↗
48. Social media marketing of IT service companies: Analysis using a concept-linking mining approach10.1016/j.indmarman.2019.11.014 ↗
49. Analyzing the trend of O2O commerce by bilingual text mining on social media10.1016/j.chb.2018.09.031 ↗
50. Embodied Human Computer Interaction10.1007/s13218-021-00727-5 ↗
51. Gesture recognition based on multi-modal feature weight10.1002/cpe.5991 ↗
52. Deep learning-based human motion recognition for predictive context-aware human-robot collaboration10.1016/j.cirp.2018.04.066 ↗
53. Deep memory network with Bi-LSTM for personalized context-aware citation recommendation10.1016/j.neucom.2020.05.047 ↗
54. Attention-based dynamic user modeling and Deep Collaborative filtering recommendation10.1016/j.eswa.2021.116036 ↗
55. Mapping Human–Computer Interaction Research Themes and Trends from Its Existence to Today: A Topic Modeling-Based Review of past 60 Years10.1080/10447318.2020.1819668 ↗
56. Federated Learning Meets Human Emotions: A Decentralized Framework for Human–Computer Interaction for IoT Applications10.1109/JIOT.2020.3037207 ↗
57. A review on human-computer interaction and intelligent robots10.1142/S0219622019300052 ↗
58. Online Hybrid CTC/Attention End-to-End Automatic Speech Recognition Architecture10.1109/TASLP.2020.2987752 ↗
59. Formosa Speech in the Wild Corpus for Improving Taiwanese Mandarin Speech-Enabled Human-Computer Interaction10.1007/s11265-019-01483-4 ↗
60. Multimodal Approach of Speech Emotion Recognition Using Multi-Level Multi-Head Fusion Attention-Based Recurrent Neural Network10.1109/ACCESS.2020.2984368 ↗
61. Hazer-Rau, D., Meudt, S., Daucher, A., Spohrs, J., Hoffmann, H., Schwenker, F., and Traue, H.C. (2020). The uulmMAC Database—A Multimodal Affective Corpus for Affective Computing in Human-Computer Interaction. Sensors, 20.10.3390/s20082308 ↗
62. Mini-batch sample selection strategies for deep learning based speech recognition10.1016/j.apacoust.2020.107573 ↗
63. Improving Low-Resource Speech Recognition Based on Improved NN-HMM Structures10.1109/ACCESS.2020.2988365 ↗
64. Leveraging Linguistic Context in Dyadic Interactions to Improve Automatic Speech Recognition for Children10.1016/j.csl.2020.101101 ↗
65. Complicated robot activity recognition by quality-aware deep reinforcement learning10.1016/j.future.2020.11.017 ↗
66. Oudah, M., Al-Naji, A., and Chahl, J. (2020). Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. J. Imaging, 6.10.3390/jimaging6080073 ↗
67. A survey on video-based Human Action Recognition: Recent updates, datasets, challenges, and applications10.1007/s10462-020-09904-8 ↗
68. The Progress of Human Pose Estimation: A Survey and Taxonomy of Models Applied in 2D Human Pose Estimation10.1109/ACCESS.2020.3010248 ↗
69. Design of hand gesture recognition system for human-computer interaction10.1007/s11042-019-08274-w ↗
70. PGCN-TCA: Pseudo Graph Convolutional Network With Temporal and Channel-Wise Attention for Skeleton-Based Action Recognition10.1109/ACCESS.2020.2964115 ↗
71. Gesture recognition algorithm based on multi-scale feature fusion in RGB-D images10.1049/iet-ipr.2020.0148 ↗
72. SGM-Net: Skeleton-guided multimodal network for action recognition10.1016/j.patcog.2020.107356 ↗
73. A framework of human action recognition using length control features fusion and weighted entropy-variances based feature selection10.1016/j.imavis.2020.104090 ↗
74. Graph convolutional network with structure pooling and joint-wise channel attention for action recognition10.1016/j.patcog.2020.107321 ↗
75. Exploring a rich spatial–temporal dependent relational model for skeleton-based action recognition by bidirectional LSTM-CNN10.1016/j.neucom.2020.07.068 ↗
76. STA-CNN: Convolutional Spatial-Temporal Attention Learning for Action Recognition10.1109/TIP.2020.2984904 ↗
77. Vision-based human action recognition: An overview and real world challenges
78. Two-Stream Convolutional Neural Network for Video Action Recognition
79. A two-fold transformation model for human action recognition using decisive pose10.1016/j.cogsys.2019.12.004 ↗
80. Tran, D.-S., Ho, N.-H., Yang, H.-J., Baek, E.-T., Kim, S.-H., and Lee, G. (2020). Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network. Appl. Sci., 10.10.3390/app10020722 ↗
81. Chen, W., Yu, C., Tu, C., Lyu, Z., Tang, J., Ou, S., Fu, Y., and Xue, Z. (2020). A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods. Sensors, 20.10.3390/s20041074 ↗
82. Human action recognition with deep learning and structural optimization using a hybrid heuristic algorithm10.1007/s10586-020-03050-0 ↗
83. Hand-crafted and deep convolutional neural network features fusion and selection strategy: An application to intelligent human action recognition10.1016/j.asoc.2019.105986 ↗
84. User Representations in Human-Computer Interaction10.1080/07370024.2020.1724790 ↗
85. DeepArSLR: A Novel Signer-Independent Deep Learning Framework for Isolated Arabic Sign Language Gestures Recognition10.1109/ACCESS.2020.2990699 ↗
86. Deep Learning Models for Real-time Human Activity Recognition with Smartphones10.1007/s11036-019-01445-x ↗
87. Pre-trained models for natural language processing: A survey10.1007/s11431-020-1647-3 ↗
88. State of art for semantic analysis of natural language processing10.48161/qaj.v1n2a40 ↗
89. A Survey of the Usages of Deep Learning for Natural Language Processing10.1109/TNNLS.2020.2979670 ↗
90. Exploring the potential of natural language processing to support microgenetic analysis of collaborative learning discussions10.1111/bjet.12875 ↗
91. NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries10.1109/TVCG.2020.3030378 ↗
92. Alexakis, G., Panagiotakis, S., Fragkakis, A., Markakis, E., and Vassilakis, K. (2019). Control of Smart Home Operations Using Natural Language Processing, Voice Recognition and IoT Technologies in a Multi-Tier Architecture. Designs, 3.10.3390/designs3030032 ↗
93. Domain-specific language model pretraining for biomedical natural language processing10.1145/3458754 ↗
94. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond10.1162/tacl_a_00511 ↗
95. Natural language processing (NLP) in management research: A literature review
96. Adversarial attacks on deep-learning models in natural language processing: A survey
97. Natural language processing for EHR-based computational phenotyping10.1109/TCBB.2018.2849968 ↗
98. English Education Game using Non-Player Character Based on Natural Language Processing10.1016/j.procs.2019.11.158 ↗
99. Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robot10.1016/j.robot.2022.104183 ↗
100. Integration of computer vision and natural language processing in multimedia robotics application