A Systematic Literature Review of Modalities, Trends, and Limitations in Emotion Recognition, Affective Computing, and Sentiment Analysis
Source abstract
Abstract
This systematic literature review delves into the extensive landscape of emotion recognition, sentiment analysis, and affective computing, analyzing 609 articles. Exploring the intricate relationships among these research domains, and leveraging data from four well-established sources—IEEE, Science Direct, Springer, and MDPI—this systematic review classifies studies in four modalities based on the types of data analyzed. These modalities are unimodal, multi-physical, multi-physiological, and multi-physical–physiological. After the classification, key insights about applications, learning models, and data sources are extracted and analyzed. This review highlights the exponential growth in studies utilizing EEG signals for emotion recognition, and the potential of multimodal approaches combining physical and physiological signals to enhance the accuracy and practicality of emotion recognition systems. This comprehensive overview of research advances, emerging trends, and limitations from 2018 to 2023 underscores the importance of continued exploration and interdisciplinary collaboration in these rapidly evolving fields.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Sentiment Analysis and Opinion Mining · Color perception and design
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Laboratorio de Tecnologías Interactivas y Experiencia de Usuario, Universidad Autónoma de Zacatecas, Jardín Juárez 147, Centro, Zacatecas 98000, Mexico
Departamento Estudios Generales, Universidad Continental, Arequipa 04001, Peru
Vicerrectorado de Investigación, Universidad Católica de Santa María, Arequipa 04001, Peru
References (67)
Show source-provided references
1. Zhou, T.H., Liang, W., Liu, H., Wang, L., Ryu, K.H., and Nam, K.W. (2022). EEG Emotion Recognition Applied to the Effect Analysis of Music on Emotion Changes in Psychological Healthcare. Int. J. Environ. Res. Public Health, 20.10.3390/ijerph20010378 ↗
2. Speech Emotion Recognition and Text Sentiment Analysis for Financial Distress Prediction10.1007/s00521-023-08470-8 ↗
3. Automated Analysis and Prediction of Job Interview Performance10.1109/TAFFC.2016.2614299 ↗
4. Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems10.1007/s40846-019-00505-7 ↗
5. Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, D.P., Murugappan, M., Chakole, Y., and Acharya, U.R. (2022). Automated Emotion Recognition: Current Trends and Future Perspectives. Comput. Methods Programs Biomed., 215.10.1016/j.cmpb.2022.106646 ↗
6. Spatio-Temporal Encoder-Decoder Fully Convolutional Network for Video-Based Dimensional Emotion Recognition10.1109/TAFFC.2019.2940224 ↗
7. Montero Quispe, K.G., Utyiama, D.M.S., dos Santos, E.M., Oliveira, H.A.B.F., and Souto, E.J.P. (2022). Applying Self-Supervised Representation Learning for Emotion Recognition Using Physiological Signals. Sensors, 22.10.3390/s22239102 ↗
8. A Multitask Learning Model for Multimodal Sarcasm, Sentiment and Emotion Recognition in Conversations10.1016/j.inffus.2023.01.005 ↗
9. Facial Expression and Body Gesture Emotion Recognition: A Systematic Review on the Use of Visual Data in Affective Computing10.1016/j.cosrev.2023.100545 ↗
10. Adapting Software with Affective Computing: A Systematic Review10.1109/TAFFC.2019.2902379 ↗
11. Deep Learning for Affective Computing: Text-Based Emotion Recognition in Decision Support10.1016/j.dss.2018.09.002 ↗
12. Ab. Aziz, N.A., K., T., Ismail, S.N.M.S., Hasnul, M.A., Ab. Aziz, K., Ibrahim, S.Z., Abd. Aziz, A., and Raja, J.E. (2023). Asian Affective and Emotional State (A2ES) Dataset of ECG and PPG for Affective Computing Research. Algorithms, 16.10.3390/a16030130 ↗
13. Utilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing10.1109/TAFFC.2019.2916015 ↗
14. Shah, S., Ghomeshi, H., Vakaj, E., Cooper, E., and Mohammad, R. (2023). An Ensemble-Learning-Based Technique for Bimodal Sentiment Analysis. Big Data Cogn. Comput., 7.10.20944/preprints202303.0460.v1 ↗
15. Tang, J., Hou, M., Jin, X., Zhang, J., Zhao, Q., and Kong, W. (2023). Tree-Based Mix-Order Polynomial Fusion Network for Multimodal Sentiment Analysis. Systems, 11.10.3390/systems11010044 ↗
16. Khamphakdee, N., and Seresangtakul, P. (2023). An Efficient Deep Learning for Thai Sentiment Analysis. Data, 8.10.3390/data8050090 ↗
17. Jo, A.-H., and Kwak, K.-C. (2023). Speech Emotion Recognition Based on Two-Stream Deep Learning Model Using Korean Audio Information. Appl. Sci., 13.10.3390/app13042167 ↗
18. Abdulrahman, A., Baykara, M., and Alakus, T.B. (2022). A Novel Approach for Emotion Recognition Based on EEG Signal Using Deep Learning. Appl. Sci., 12.10.3390/app121910028 ↗
19. Deep Learning Based Multimodal Emotion Recognition Using Model-Level Fusion of Audio–Visual Modalities10.1016/j.knosys.2022.108580 ↗
20. Ali, M., Mosa, A.H., Al Machot, F., and Kyamakya, K. (2016, January 5–8). EEG-Based Emotion Recognition Approach for e-Healthcare Applications. Proceedings of the 2016 Eighth International Conference on Ubiquitous and Future Networks (ICUFN), Vienna, Austria.10.1109/ICUFN.2016.7536936 ↗
21. Driver Emotion Recognition for Intelligent Vehicles10.1145/3388790 ↗
22. A Novel Driver Emotion Recognition System Based on Deep Ensemble Classification10.1007/s40747-023-01100-9 ↗
23. Human Emotion Recognition for Enhanced Performance Evaluation in E-Learning10.1007/s13748-022-00278-2 ↗
24. Alaei, A., Wang, Y., Bui, V., and Stantic, B. (2023). Target-Oriented Data Annotation for Emotion and Sentiment Analysis in Tourism Related Social Media Data. Future Internet, 15.10.3390/fi15040150 ↗
25. Caratù, M., Brescia, V., Pigliautile, I., and Biancone, P. (2023). Assessing Energy Communities’ Awareness on Social Media with a Content and Sentiment Analysis. Sustainability, 15.10.3390/su15086976 ↗
26. A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals10.1109/ACCESS.2019.2944001 ↗
27. Emotion Recognition from Physiological Signal Analysis: A Review10.1016/j.entcs.2019.04.009 ↗
28. Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., and Yang, X. (2018). A Review of Emotion Recognition Using Physiological Signals. Sensors, 18.10.3390/s18072074 ↗
29. A Survey on Facial Emotion Recognition Techniques: A State-of-the-Art Literature Review10.1016/j.ins.2021.10.005 ↗
30. Ubiquitous Affective Computing: A Review10.1109/JSEN.2021.3138269 ↗
31. Schmidt, P., Reiss, A., Dürichen, R., and Laerhoven, K. (2019). Van Wearable-Based Affect Recognition—A Review. Sensors, 19.10.3390/s19194079 ↗
32. Deep Learning for Human Affect Recognition: Insights and New Developments10.1109/TAFFC.2018.2890471 ↗
33. A Systematic Survey on Multimodal Emotion Recognition Using Learning Algorithms
34. Kitchenham, B. (2004). Procedures for Performing Systematic Reviews, Keele University.
35. AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild10.1109/TAFFC.2017.2740923 ↗
36. Video-Based Depression Level Analysis by Encoding Deep Spatiotemporal Features10.1109/TAFFC.2018.2870884 ↗
37. Exploiting Multi-CNN Features in CNN-RNN Based Dimensional Emotion Recognition on the OMG in-the-Wild Dataset10.1109/TAFFC.2020.3014171 ↗
38. A Deeper Look at Facial Expression Dataset Bias10.1109/TAFFC.2020.2973158 ↗
39. Automatic Recognition of Facial Displays of Unfelt Emotions10.1109/TAFFC.2018.2874996 ↗
40. Punuri, S.B., Kuanar, S.K., Kolhar, M., Mishra, T.K., Alameen, A., Mohapatra, H., and Mishra, S.R. (2023). Efficient Net-XGBoost: An Implementation for Facial Emotion Recognition Using Transfer Learning. Mathematics, 11.10.3390/math11030776 ↗
41. Mukhiddinov, M., Djuraev, O., Akhmedov, F., Mukhamadiyev, A., and Cho, J. (2023). Masked Face Emotion Recognition Based on Facial Landmarks and Deep Learning Approaches for Visually Impaired People. Sensors, 23.10.3390/s23031080 ↗
42. Babu, E.K., Mistry, K., Anwar, M.N., and Zhang, L. (2022). Facial Feature Extraction Using a Symmetric Inline Matrix-LBP Variant for Emotion Recognition. Sensors, 22.10.3390/s22228635 ↗
43. Mustafa Hilal, A., Elkamchouchi, D.H., Alotaibi, S.S., Maray, M., Othman, M., Abdelmageed, A.A., Zamani, A.S., and Eldesouki, M.I. (2022). Manta Ray Foraging Optimization with Transfer Learning Driven Facial Emotion Recognition. Sustainability, 14.10.3390/su142114308 ↗
44. Emotion Recognition at a Distance: The Robustness of Machine Learning Based on Hand-Crafted Facial Features vs Deep Learning Models10.1016/j.imavis.2023.104724 ↗
45. Deep Learning-Based Dimensional Emotion Recognition Combining the Attention Mechanism and Global Second-Order Feature Representations10.1016/j.compeleceng.2022.108469 ↗
46. On-Road Driver Facial Expression Emotion Recognition with Parallel Multi-Verse Optimizer (PMVO) and Optical Flow Reconstruction for Partial Occlusion in Internet of Things (IoT)10.1016/j.measen.2023.100711 ↗
47. Emotion Recognition by Web-Shaped Model10.1007/s11042-022-13361-6 ↗
48. Edge-Enhanced Bi-Dimensional Empirical Mode Decomposition-Based Emotion Recognition Using Fusion of Feature Set10.1007/s00500-016-2395-4 ↗
49. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13–18). The Extended Cohn-Kanade Dataset (CK+): A Complete Dataset for Action Unit and Emotion-Specified Expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543262 ↗
50. Facial Expression Recognition from Near-Infrared Videos10.1016/j.imavis.2011.07.002 ↗
51. Barros, P., Churamani, N., Lakomkin, E., Siqueira, H., Sutherland, A., and Wermter, S. (2018, January 8–13). The OMG-Emotion Behavior Dataset. Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.10.1109/IJCNN.2018.8489099 ↗
52. Improved Deep CNN-Based Two Stream Super Resolution and Hybrid Deep Model-Based Facial Emotion Recognition10.1016/j.engappai.2022.105486 ↗
53. Cross-Domain Color Facial Expression Recognition Using Transductive Transfer Subspace Learning10.1109/TAFFC.2016.2563432 ↗
54. Tan, K.L., Lee, C.P., and Lim, K.M. (2023). RoBERTa-GRU: A Hybrid Deep Learning Model for Enhanced Sentiment Analysis. Appl. Sci., 13.10.3390/app13063915 ↗
55. Multi-Loop Graph Convolutional Network for Multimodal Conversational Emotion Recognition10.1016/j.jvcir.2023.103846 ↗
56. Multi-Fusion Residual Memory Network for Multimodal Human Sentiment Comprehension10.1109/TAFFC.2020.3000510 ↗
57. Integrating Deep and Shallow Models for Multi-Modal Depression Analysis—Hybrid Architectures10.1109/TAFFC.2018.2870398 ↗
58. Multimodal Emotion Recognition Using Cross Modal Audio-Video Fusion with Attention and Deep Metric Learning10.1016/j.imavis.2023.104676 ↗
59. Audio-Visual Emotion Recognition in Video Clips10.1109/TAFFC.2017.2713783 ↗
60. SAMM: A Spontaneous Micro-Facial Movement Dataset10.1109/TAFFC.2016.2573832 ↗
61. Fuzzy Histogram of Optical Flow Orientations for Micro-Expression Recognition10.1109/TAFFC.2017.2723386 ↗
62. Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., and Van Laerhoven, K. (2018, January 2). Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection. Proceedings of the Proceedings of the 20th ACM International Conference on Multimodal Interaction, New York, NY, USA.10.1145/3242969.3242985 ↗
63. AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups10.1109/TAFFC.2018.2884461 ↗
64. ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors10.1109/TAFFC.2016.2625250 ↗
65. DEAP: A Database for Emotion Analysis; Using Physiological Signals10.1109/T-AFFC.2011.15 ↗
66. Zhang, Y., Cheng, C., Wang, S., and Xia, T. (2022). Emotion Recognition Using Heterogeneous Convolutional Neural Networks Combined with Multimodal Factorized Bilinear Pooling. Biomed. Signal Process Control, 77.10.1016/j.bspc.2022.103877 ↗
67. (2024, August 12). The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews. Available online: https://www.prisma-statement.org/prisma-2020-statement.