Classroom Behavior Detection Based on Improved YOLOv5 Algorithm Combining Multi-Scale Feature Fusion and Attention Mechanism
Source abstract
Abstract
The detection of students’ behaviors in classroom can provide a guideline for assessing the effectiveness of classroom teaching. This study proposes a classroom behavior detection algorithm using an improved object detection model (i.e., YOLOv5). First, the feature pyramid structure (FPN+PAN) in the neck network of the original YOLOv5 model is combined with a weighted bidirectional feature pyramid network (BiFPN). They are subsequently processed with feature fusion of different scales of the object to mine the fine-grained features of different behaviors. Second, a spatial and channel convolutional attention mechanism (CBAM) is added between the neck network and the prediction network to make the model focus on the object information to improve the detection accuracy. Finally, the original non-maximum suppression is improved using the distance-based intersection ratio (DIoU) to improve the discrimination of occluded objects. A series of experiments were conducted on our new established dataset which includes four types of behaviors: listening, looking down, lying down, and standing. The results demonstrated that the algorithm proposed in this study can accurately detect various student behaviors, and the accuracy was higher than that of the YOLOv5 model. By comparing the effects of student behavior detection in different scenarios, the improved algorithm had an average accuracy of 89.8% and a recall of 90.4%, both of which were better than the compared detection algorithms.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Surveillance and Tracking Methods · Human Pose and Action Recognition · Fire Detection and Safety Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer and Information Science, Chongqing Normal University, Chongqing 401331, China
Faculty of Education, Southwest University, Chongqing 400715, China
Funding
Chongqing Educational Planning Project · Chongqing Education Commission Science and Technology Research Project · Teaching Reform Research Project of Chongqing Higher Education
References (40)
Show source-provided references
1. A research framework of smart education10.1186/s40561-016-0026-2 ↗
2. The instructional process: A review of Flanders’ interaction analysis in a classroom setting10.11648/j.ijsedu.20150305.11 ↗
3. Reducing the dimensionality of data with neural networks10.1126/science.1127647 ↗
4. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
5. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
6. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
7. Paszkiel, S., and Dobrakowski, P. (2021). The use of multilayer convnets for the purposes of motor imagery classification. Conference on Automation, Springer.10.1007/978-3-030-74893-7_2 ↗
8. Two-Stream Convolutional Neural Network for Video Action Recognition
9. Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., and Gool, L.V. (2016, January 8–16). Temporal segment networks: Towards good practices for deep action recognition. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_2 ↗
10. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.510 ↗
11. Hara, K., Kataoka, H., and Satoh, Y. (2018, January 18–23). Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00685 ↗
12. Qiu, Z., Yao, T., and Mei, T. (2017, January 22–29). Learning spatio-temporal representation with pseudo-3d residual networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.590 ↗
13. Cao, Z., Simon, T., Wei, S.-E., and Sheikh, Y. (2017, January 21–26). Realtime multi-person 2d pose estimation using part affinity fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
14. Pose detection in complex classroom environment based on improved Faster R-CNN10.1049/iet-ipr.2018.5905 ↗
15. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
16. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 24–27). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
17. Spatial pyramid pooling in deep convolutional networks for visual recognition10.1109/TPAMI.2015.2389824 ↗
18. Girshick, R. (2015, January 7–13). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiage, Chile.10.1109/ICCV.2015.169 ↗
19. Faster r-cnn: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
20. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
21. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A.C. (2016, January 8–16). Ssd: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
22. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
23. Mota, S., and Picard, R.W. (2003, January 16–22). Automated posture analysis for detecting learner’s interest level. Proceedings of the 2003 Conference on Computer Vision and Pattern Recognition Workshop, Madison, WI, USA.10.1109/CVPRW.2003.10047 ↗
24. Students activity recognition by heart rate monitoring in classroom using k-means classification10.20473/jisebi.6.1.46-54 ↗
25. Student body gesture recognition based on Fisher broad learning system10.1142/S0219691319500012 ↗
26. Lin, F.-C., Ngo, H.-H., Dow, C.-R., Lam, K.-H., and Le, H.L. (2021). Student Behavior Recognition System for the Classroom Environment Based on Skeleton Pose Estimation and Person Detection. Sensors, 21.10.3390/s21165314 ↗
27. An Automatic Recognition Method for Students’ Classroom Behaviors Based on Image Processing10.18280/ts.370318 ↗
28. Recognition of Student Classroom Behaviors Based on Moving Target Detection10.18280/ts.380123 ↗
29. Liu, H., Ao, W., and Hong, J. (2021, January 22–24). Student Abnormal Behavior Recognition in Classroom Video Based on Deep Learning. Proceedings of the 2021 5th International Conference on Electronic Information Technology and Computer Engineering, Xiamen, China.10.1145/3501409.3501529 ↗
30. Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv.
31. Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv.
32. Wang, C.Y., Liao, H.Y.M., Wu, Y.H., Chen, P.Y., Hsieh, J.W., and Yeh, I.H. (2020, January 14–16). CSPNet: A new backbone that can enhance learning capability of CNN. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00203 ↗
33. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
34. Liu, S., Qi, L., Qin, H., Shi, J., and Jia, J. (2018, January 18–23). Path aggregation network for instance segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00913 ↗
35. Tan, M., Pang, R., and Le, Q.V. (2020, January 13–19). Efficientdet: Scalable and efficient object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01079 ↗
36. Liu, S., Huang, D., and Wang, Y. (2019). Learning spatial fusion for single-shot object detection. arXiv.
37. Woo, S., Park, J., Lee, J.-Y., and Kweon, I.S. (2018, January 8–14). Cbam: Convolutional block attention module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
38. Gong, M., Wang, D., Zhao, X., Guo, H., Luo, D., and Song, M. (2021). A review of non-maximum suppression algorithms for deep learning target detection. Seventh Symposium on Novel Photoelectronic Detection Technology and Applications, SPIE.10.1117/12.2586477 ↗
39. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
40. An anatomization on breast cancer detection and diagnosis employing multi-layer perceptron neural network (MLP) and Convolutional neural network (CNN)10.1016/j.ceh.2020.11.002 ↗