journal article·2023-02-17Open access

Deep Analysis of Student Body Activities to Detect Engagement State in E-Learning Sessions

Shoroog Ghazee Khenkar, Salma Kammoun Jarraya, Arwa Allinjawi, Samar Alkhuraiji, Nihal Abuzinadah, Faris A. Kateb

Applied Sciences· Volume 13· Issue 4

Abstract

In this paper, we propose new 3D CNN prediction models for detecting student engagement levels in an e-learning environment. The first generated model classifies students’ engagement to high positive engagement or low positive engagement. The second generated model classifies engagement to low negative engagement or disengagement. To predict the engagement level, the proposed prediction models learn the deep spatiotemporal features of the body activities of the students. In addition, we collected a new video dataset for this study. The new dataset was collected in realistic, uncontrolled settings from real students attending real online classes. Our findings are threefold: (1) Spatiotemporal features are more suitable for analyzing body activities from video data; (2) our proposed prediction models outperform state-of-the-art methods and have proven their effectiveness; and (3) our newly collected video dataset, which reflects realistic scenarios, contributed to delivering comparable results to current methods. The findings of this work will strengthen the knowledge base for the development of intelligent and interactive e-learning systems that can give feedback based on user engagement.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Online Learning and Analytics · Mind wandering and attention · Mental Health Research Topics

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Computer Science, King Abdulaziz University, Jeddah 22254, Saudi Arabia

Department of Information Technology, King Abdulaziz University, Jeddah 22254, Saudi Arabia

Funding

Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia · King Abdulaziz University, DSR, Jeddah, Saudi Arabia

References (29)

Show source-provided references
1. Ricardo, M.L., and Iglesias, M.J. (2018, January 13–16). Analyzing learners’ experience in e-learning based scenarios using intelligent alerting systems: Awakening of new and improved solutions. Proceedings of the 13th Iberian Conference on Information Systems and Technologies (CISTI 2018), Cáceres, Spain.
2. Affective Computing: A Review10.1007/11573548_125 ↗
3. Affective e-learning: Using “emotional” data to improve learning in pervasive learning environment
4. Moubayed, A., Injadat, M., Shami, A., and Lutfiyya, H. (2018, January 11–14). Relationship between student engagement and performance in e-learning environment using association rules. Proceedings of the IEEE World Engineering Education Conference (EDUNINE 2018), Buenos Aires, Argentina.10.1109/EDUNINE.2018.8451005 ↗
5. Automated detection of engagement using video-based estimation of facial expressions and heart rate10.1109/TAFFC.2016.2515084 ↗
6. Jang, M., Park, C., Yang, H.S., Kim, Y.H., Cho, Y.J., Lee, D.W., Cho, H.K., Kim, Y.A., Chae, K., and Ahn, B.K. (2014, January 3–6). Building an automated engagement recognizer based on video analysis. Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI 14), Bielefeld, Germany.10.1145/2559636.2563687 ↗
7. Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics10.1145/3015297.3015301 ↗
8. A new emotion–based affective model to detect student’s engagement
9. Engagement detection based on analysing micro body gestures using 3d cnn
10. Multimodal student engagement recognition in prosocial games10.1109/TCIAIG.2017.2743341 ↗
11. Assessing learning engagement based on facial expression recognition in MOOC’s scenario10.1007/s00530-021-00854-x ↗
12. Li, Y.Y., and Hung, Y.P. (2019, January 22–29). Feature fusion of face and body for engagement intensity detection. Proceedings of the IEEE International Conference on Image Processing (ICIP 2019), Taipei, Taiwan.10.1109/ICIP.2019.8803488 ↗
13. Dhall, A., Kaur, A., Goecke, R., and Gedeon, T. (2018, January 16–20). Emotiw2018: Audio-video, student engagement and group-level affect prediction. Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI 18), Boulder, CO, USA.10.1145/3242969.3264993 ↗
14. Dash, S., Dewan, M.A., Murshed, M., Lin, F., Abdullah, M., and Das, A. (2019, January 24–25). A two-stage algorithm for engagement detection in online learning. Proceedings of the International Conference on Sustainable Technologies for Industry 4.0 (STI), Dhaka, Bangladesh.10.1109/STI47673.2019.9068054 ↗
15. Gupta, A., Jaiswal, R., Adhikari, S., and Balasubramanian, V. (2016). DAISEE: Dataset for affective states in e-learning environments. arXiv.
16. Engagement detection in online learning: A review10.1186/s40561-018-0080-z ↗
17. Chen, L., and Nugent, C.D. (2019). Human Activity Recognition and Behaviour Analysis, Springer International Publishing. [1st ed.].10.1007/978-3-030-19408-6_7 ↗
18. Darwin, C. (1872). The Expression of the Emotions in Man and Animals, John Murray.10.1037/10001-000 ↗
19. Affective body expression perception and recognition: A survey10.1109/T-AFFC.2012.16 ↗
20. Argyle, M. (1988). Bodily Communication, Routledge.
21. Body expressions influence recognition of emotions in the face and voice10.1037/1528-3542.7.3.487 ↗
22. Detecting deception from the body or face10.1037/h0036006 ↗
23. Associating facial expressions and upper-body gestures with learning tasks for enhancing intelligent tutoring systems10.1007/s40593-020-00195-2 ↗
24. Identifying features of bodily expression as indicators of emotional experience during multimedia learning10.3389/fpsyg.2017.01303 ↗
25. Lie catching and microexpressions
26. Tran, D., Bourdev, L.D., Fergus, L., and Paluri, M. (2014). C3D: Generic features for video analysis. arXiv.
27. The faces of engagement: Automatic recognition of student engagement from facial expressions10.1109/TAFFC.2014.2316163 ↗
28. Nezami, O.M., Dras, M.L., Richards, D., Wan, S., and Paris, C. (2020, January 16–20). Automatic recognition of student engagement using deep learning and facial expression. Proceedings of the 19th Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), Wurzburg, Germany.
29. Dewan, M.A., Lin, F., Wen, D., Murshed, M., and Uddin, Z. (2018, January 8–12). A deep learning approach to detecting engagement of online learners. Proceedings of the 2018 IEEE SmartWorld, Ubiquitous Intelligence Computing, Advanced Trusted Computing, Scalable Computing Communications, Cloud Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), Los Alamitos, CA, USA.10.1109/SmartWorld.2018.00318 ↗