Research on Students’ Action Behavior Recognition Method Based on Classroom Time-Series Images
Source abstract
Abstract
Students’ action behavior performance is an important part of classroom teaching evaluation. To detect the action behavior of students in classroom teaching videos, and based on the detection results, the action behavior sequence of individual students in the teaching time of knowledge points is obtained and analyzed. This paper proposes a method for recognizing students’ action behaviors based on classroom time-series images. First, we propose an improved Asynchronous Interaction Aggregation (AIA) network for student action behavior detection. By adding a Multi-scale Temporal Attention (MsTA) module and a Multi-scale Channel Spatial Attention (MsCSA) module to the fast pathway and slow pathway, respectively, the accuracy of student action behavior recognition is improved in SlowFast, which is the backbone network of the improved AIA network,. Second, the Equalized Focal Loss function is introduced to improve the category imbalance that exists in the student action behavior dataset. Experimental results on the student action behavior dataset show that the method proposed in this paper can detect different action behaviors of students in the classroom and has better detection performance compared to the original AIA network. Finally, based on the results of action behavior recognition, the seat number is used as the index to obtain the action behavior sequence of individual students during the teaching time of knowledge points and the performance of students in this period is analyzed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Anomaly Detection Techniques and Applications · Human Pose and Action Recognition · AI and Big Data Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Communication, Guilin University of Electronic Technology, Guilin 541004, China
School of Computer and Information Security, Guilin University of Electronic Technology, Guilin 541004, China
Funding
The National Natural Science Foundation of China
References (42)
Show source-provided references
1. Fu, R., Wu, T., Luo, Z., Duan, F., Qiao, X., Guo, P., and IEEE (2019, January 14–19). Learning Behavior Analysis in Classroom Based on Deep Learning. Proceedings of the 10th International Conference on Intelligent Control and Information Processing, Marrakesh, Morocco.10.1109/ICICIP47338.2019.9012177 ↗
2. Zhang, Y., and Wang, G. (2021, January 23–25). Research on Application of Intelligent Analysis in Monitoring of Classroom Teaching. Proceedings of the 2021 3rd International Conference on Advances in Computer Technology, Information Science and Communication, Shanghai, China.10.1109/CTISC52352.2021.00053 ↗
3. A Framework of Adaptive Multiscale Wavelet Decomposition for Signals on Undirected Graphs10.1109/TSP.2019.2896246 ↗
4. Hyperspectral image classification using wavelet transform-based smooth ordering10.1142/S0219691319500504 ↗
5. Introducing the Discrete Path Transform (DPT) and its applications in signal analysis, artefact removal, and spoken word recognition10.1016/j.dsp.2021.103158 ↗
6. Guariglia, E. (2019). Primality, Fractality, and Image Analysis. Entropy, 21.10.3390/e21030304 ↗
7. Bui Ngoc, A., Ngo Tung, S., Phan Truong, L., Le Phuong, C., Nguyen Huu, T., Nguyen Cong, D., Nguyen Huu, T., Aftab, M.U., and Tran Van, D. (2019). A Computer-Vision Based Application for Student Behavior Monitoring in Classroom. Appl. Sci., 9.10.3390/app9224729 ↗
8. Pose detection in complex classroom environment based on improved Faster R-CNN10.1049/iet-ipr.2018.5905 ↗
9. An improved method of identifying learner’s behaviors based on deep learning10.1007/s11227-022-04402-w ↗
10. Abnormal Behavior Recognition in Classroom Pose Estimation of College Students Based on Spatiotemporal Representation Learning10.18280/ts.380109 ↗
11. Tang, J., Xia, J., Mu, X., Pang, B., and Lu, C. (2020, January 23–28). Asynchronous Interaction Aggregation for Action Detection. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58555-6_5 ↗
12. Li, B., Yao, Y., Tan, J., Zhang, G., Yu, F., Lu, J., and Luo, Y. (2022, January 18–24). Equalized Focal Loss for Dense Long-Tailed Object Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00686 ↗
13. Feichtenhofer, C., Fan, H., Malik, J., and He, K. (November, January 27). SlowFast Networks for Video Recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
14. Feichtenhofer, C. (2020, January 14–19). X3D: Expanding Architectures for Efficient Video Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00028 ↗
15. Chang, S., Wang, P., Wang, F., Feng, J., and Shou, M.Z. (2023, January 17–24). DOAD: Decoupled One Stage Action Detection Network. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Vancouver, BC, Canada.10.1109/CVPRW59228.2023.00314 ↗
16. Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., and Feichtenhofer, C. (2021, January 11–17). Multiscale Vision Transformers. Proceedings of the 18th IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00675 ↗
17. Pan, J., Chen, S., Shou, M.Z., Liu, Y., Shao, J., and Li, H. (2021, January 19–25). Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.00053 ↗
18. Zheng, Y.D., Chen, G., Yuan, M., and Lu, T. (2023, January 10–14). MRSN: Multi-Relation Support Network for Video Action Detection. Proceedings of the 2023 IEEE International Conference on Multimedia and Expo, Brisbane, Australia.10.1109/ICME55011.2023.00180 ↗
19. Faure, G.J., Chen, M.H., and Lai, S.H. (2023, January 2–7). Holistic Interaction Transformer Network for Action Detection. Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV56688.2023.00334 ↗
20. Classroom student posture recognition based on an improved high-resolution network10.1186/s13638-021-02015-0 ↗
21. Classroom Learning Status Assessment Based on Deep Learning
22. Simulation of student classroom behavior recognition based on cluster analysis and random forest algorithm10.3233/JIFS-189237 ↗
23. An Intelligent System for Detecting Abnormal Behavior in Students Based on the Human Skeleton and Deep Learning10.1155/2022/3819409 ↗
24. Recognition of Student Classroom Behaviors Based on Moving Target Detection10.18280/ts.380123 ↗
25. Multimodal behavior analysis in computer-enabled laboratories using nonverbal cues10.1007/s11760-020-01705-4 ↗
26. Behaviour detection in crowded classroom scenes via enhancing features robust to scale and perspective variations10.1049/ipr2.12318 ↗
27. An Automatic Recognition Method for Students’ Classroom Behaviors Based on Image Processing10.18280/ts.370318 ↗
28. Zheng, Z., Liang, G., Luo, H., and Yin, H. (2022). Attention assessment based on multi-view classroom behaviour recognition. IET Comput. Vis.10.1049/cvi2.12146 ↗
29. Student Behavior Recognition From Heterogeneous View Perception in Class Based on 3-D Multiscale Residual Dense Network for the Analysis of Case Teaching10.3389/fnbot.2021.675827 ↗
30. A new feature fusion network for student behavior recognition in education
31. He, K., Gkioxari, G., Dollar, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the 16th IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
32. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
33. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention Is All You Need. Proceedings of the 31st Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA.
34. Chen, Y., Kalantidis, Y., Li, J., Yan, S., and Feng, J. (2018, January 2–8). A2-Nets: Double Attention Networks. Proceedings of the 32nd Conference on Neural Information Processing Systems, Montreal, QC, Canada.
35. Li, Y., Chen, Y., Wang, N., Zhang, Z., and IEEE (November, January 27). Scale-Aware Trident Networks for Object Detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
36. Ruan, D., Wang, D., Zheng, Y., Zheng, N., and Zheng, M. (2021, January 19–25). Gaussian Context Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01488 ↗
37. Focal Loss for Dense Object Detection10.1109/TPAMI.2018.2858826 ↗
38. Zheng, Z., Wang, P., Liu, W., Li, J., Ye, R., Ren, D., and Assoc Advancement Artificial, I. (2020, January 7–12). Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression. Proceedings of the 34th AAAI Conference on Artificial Intelligence/32nd Innovative Applications of Artificial Intelligence Conference/10th AAAI Symposium on Educational Advances in Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6999 ↗
39. Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan, S., Toderici, G., Ricco, S., and Sukthankar, R. (2018, January 18–23). AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions. Proceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00633 ↗
40. Kalogeiton, V., Weinzaepfel, P., Ferrari, V., and Schmid, C. (2017, January 22–29). Action Tubelet Detector for Spatio-Temporal Action Localization. Proceedings of the 16th IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.472 ↗
41. Yang, X., Yang, X., Liu, M.-Y., Xiao, F., Davis, L., Kautz, J., and Soc, I.C. (2019, January 16–20). STEP: Spatio-Temporal Progressive Learning for Video Action Detection. Proceedings of the 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00035 ↗
42. Tan, J.R., Lu, X., Zhang, G., Yin, C.Q., and Li, Q.Q. (2021, January 19–25). Equalization Loss v2: A New Gradient Balance Approach for Long-tailed Object Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.00173 ↗