Collaborative Analysis of Learners’ Emotional States Based on Cross-Modal Higher-Order Reasoning
Source abstract
Abstract
Emotion is a significant factor influencing education and teaching, closely intertwined with learners’ cognitive processing. Conducting analysis of learners’ emotions based on cross-modal data is beneficial for achieving personalized guidance in intelligent educational environments. Currently, due to factors such as data scarcity and environmental noise, data imbalances have led to incomplete or missing emotional information. Therefore, this study proposes a collaborative analysis model based on attention mechanisms. The model extracts features from various types of data using different tools and employs multi-head attention mechanisms for parallel processing of feature vectors. Subsequently, through a cross-modal attention collaborative interaction module, effective interaction among visual, auditory, and textual information is facilitated, significantly enhancing comprehensive understanding and the analytical capabilities of cross-modal data. Finally, empirical evidence demonstrates that the model can effectively improve the accuracy and robustness of emotion recognition in cross-modal data.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Educational Technology and Assessment
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer, Guangdong University of Technology, Guangzhou 511400, China
College of Automation, Guangdong University of Technology, Guangzhou 511400, China
Center of Campus Network & Modern Educational Technology, Guangdong University of Technology, Guangzhou 510006, China
Funding
Guangdong Provincial Philosophy and Social Science Planning Project of China · National Natural Science Foundation of China
References (18)
Show source-provided references
1. BNU-LSVED 2.0: Spontaneous multimodal student affect database with multi-dimensional labels10.1016/j.image.2017.08.012 ↗
2. Trabelsi, Z., Alnajjar, F., Parambil, M.M.A., Gochoo, M., and Ali, L. (2023). Real-Time Attention Monitoring System for Classroom: A Deep Learning Approach for Student’s Behavior Recognition. Big Data Cogn. Comput., 7.10.3390/bdcc7010048 ↗
3. Ma, L., Lu, Z., Shang, L., and Li, H. (2015, January 7–13). Multimodal convolutional neural networks for matching image and sentence. Proceedings of the IEEE International Conference on Computer Vision, Washington, DC, USA.10.1109/ICCV.2015.301 ↗
4. A review of affective computing: From unimodal analysis to multimodal fusion10.1016/j.inffus.2017.02.003 ↗
5. Facial Action Recognition for Facial Expression Analysis From Static Face Images10.1109/TSMCB.2004.825931 ↗
6. Recognition of Facial Expression Using Gabor Wavelets and Learning Vector Quantization10.1016/j.engappai.2007.11.010 ↗
7. Natural Facial Expression Recognition Using Differential-AAM and Manifold Learning10.1016/j.patcog.2008.10.010 ↗
8. Shan, C.F., Gong, S.G., and Peter, W. (2006, January 17–22). A Comprehensive Empirical Study on Linear Subspace Methods for Facial Expression Analysis. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshop, Washington, DC, USA.
9. A review of multimodal sentiment analysis
10. Research on emotion recognition methods based on audio and video
11. Degottex, G., Kane, J., Drugman, T., Raitio, T., and Scherer, S. (2014, January 4–9). COVAREP—A collaborative voice analysis repository for speech technologies. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6853739 ↗
12. Baltrusaitis, T., Robinson, P., and Morency, L.P. (2016, January 7–10). OpenFace: An open source facial behavior analysis toolkit. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.10.1109/WACV.2016.7477553 ↗
13. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 7–10). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
14. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
15. Sahay, S., Okur, E., Kumar, S.H., and Nachman, L. (2020, January 5–10). Low rank fusion based transformers for multimodal sequences. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.challengehml-1.4 ↗
16. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. (2017, January 9–11). Tensor fusion network for multimodal sentiment analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1115 ↗
17. Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.-P. (2018, January 2–7). Memory fusion network for multi-view sequential learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12021 ↗
18. Tsai, Y.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.-P., and Salakhutdinov, R. (August, January 28). Multimodal transformer for unaligned multimodal language sequences. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.