A Lightweight Spatiotemporal Skeleton Network for Abnormal Train Driver Action Detection
Source abstract
Abstract
Abnormal behaviors of train drivers are a critical factor affecting the operational safety of urban rail transit. To achieve automated and efficient detection while meeting practical deployment requirements, this study proposes an end-to-end Temporal Action Detection network based on skeleton data. The network directly uses skeleton sequences as input, integrates a skeleton topology graph tailored to train driver actions for spatiotemporal feature extraction, and employs a non-shared feature propagation design to enhance classification and regression performance. Evaluated on a custom dataset of driver operations (including both standard and abnormal behaviors), the experimental results demonstrate favorable performance with high mean Average Precision (mAP) and strong accuracy. The findings show that the proposed network can accurately localize and classify driver operational behaviors, enabling precise detection of abnormal actions. Furthermore, its low parameter count and minimal storage requirements highlight strong potential for practical deployment in urban rail transit systems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Anomaly Detection Techniques and Applications · Human Pose and Action Recognition · Railway Systems and Energy Efficiency
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Urban Railway Transportation, Shanghai University of Engineering Science, Shanghai 201620, China
Funding
National Natural Science Foundation of China
References (69)
Show source-provided references
1. Li, X. (2024, January 8–10). Detection of Power System Personnel’s Abnormal Behavior Based on Machine Vision. Proceedings of the 2024 Boao New Power System International Forum—Power System and New Energy Technology Innovation Forum (NPSIF), Qionghai, China.10.1109/NPSIF64134.2024.10883505 ↗
2. Leibe, B., Matas, J., Sebe, N., and Welling, M. (2016). Online Action Detection. Computer Vision—ECCV 2016, Springer International Publishing.10.1007/978-3-319-46454-1 ↗
3. An, J., Kang, H., Han, S.H., Yang, M.-H., and Kim, S.J. (2023, January 2–6). MiniROAD: Minimal RNN Framework for Online Action Detection. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00949 ↗
4. Xu, M., Gao, M., Chen, Y.-T., Davis, L., and Crandall, D. (November, January 29). Temporal Recurrent Networks for Online Action Detection. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00563 ↗
5. Chen, J., Mittal, G., Yu, Y., Kong, Y., and Chen, M. (2022, January 19–24). GateHUB: Gated History Unit with Background Suppression for Online Action Detection. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01930 ↗
6. Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., and Hassner, T. (2022). Uncertainty-Based Spatial-Temporal Attention for Online Action Detection. Computer Vision—ECCV 2022, Springer Nature Switzerland.10.1007/978-3-031-20068-7 ↗
7. Huang, X., and Mao, Q. (2025). Skeleton-Based Online Action Detection with Temporal Enhancement. Emotional Intelligence, Springer Nature.10.1007/978-981-96-5084-2 ↗
8. Advancing Driver Behavior Recognition: An Intelligent Approach Utilizing ResNet10.3103/S0146411624700664 ↗
9. Iyer, B., Ghosh, D., and Balas, V.E. (2022). Driver Drowsiness Detection Using Deep Learning. Applied Information Processing Systems, Springer.10.1007/978-981-16-2008-9 ↗
10. Darapaneni, N., Arora, J., Hazra, M., Vig, N., Gandhi, S.S., Gupta, S., and Paduri, A.R. (2022). Detection of Distracted Driver Using Convolution Neural Network. arXiv.
11. Driver Behaviors Recognizer Based on Light-Weight Convolutional Neural Network Architecture and Attention Mechanism10.1109/ACCESS.2022.3187185 ↗
12. Video-Based Abnormal Driving Behavior Detection via Deep Learning Fusions10.1109/ACCESS.2019.2917213 ↗
13. Neves, A.J.R. (2019). Advance Intelligent Video Surveillance System (AIVSS): A Future Aspect. Intelligent Video Surveillance, IntechOpen.
14. Deep Learning-Based Action Detection in Untrimmed Videos: A Survey10.1109/TPAMI.2022.3193611 ↗
15. Shou, Z., Wang, D., and Chang, S.-F. (2016, January 27–30). Temporal Action Localization in Untrimmed Videos via Multi-Stage CNNs. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.119 ↗
16. Lin, T., Liu, X., Li, X., Ding, E., and Wen, S. (November, January 27). BMN: Boundary-Matching Network for Temporal Action Proposal Generation. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00399 ↗
17. Xu, H., Das, A., and Saenko, K. (2017, January 22–29). R-C3D: Region Convolutional 3D Network for Temporal Activity Detection. Proceedings of the 2017 IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy.10.1109/ICCV.2017.617 ↗
18. MBGNet: Multi-Branch Boundary Generation Network with Temporal Context Aggregation for Temporal Action Detection10.1007/s10489-024-05664-y ↗
19. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
20. Simonyan, K., and Zisserman, A. (2014). Two-Stream Convolutional Networks for Action Recognition in Videos. Proceedings of the 28th International Conference on Neural Information Processing Systems, MIT Press. [1st ed.].
21. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.510 ↗
22. Qiu, Z., Yao, T., and Mei, T. (2017, January 22–29). Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.590 ↗
23. Human Action Recognition From Various Data Modalities: A Review
24. Cao, Z., Simon, T., Wei, S.-E., and Sheikh, Y. (2017, January 21–26). Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
25. AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time10.1109/TPAMI.2022.3222784 ↗
26. Shi, L., Zhang, Y., Cheng, J., and Lu, H. (2019, January 15–20). Skeleton-Based Action Recognition With Directed Graph Neural Networks. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00810 ↗
27. InfoGCN++: Learning Representation by Predicting the Future for Online Skeleton-Based Action Recognition10.1109/TPAMI.2024.3466212 ↗
28. Li, B., Chen, H., Chen, Y., Dai, Y., and He, M. (2017, January 10–14). Skeleton Boxes: Solving Skeleton Based Action Detection with a Single Deep Convolutional Neural Network. Proceedings of the 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Hong Kong, China.
29. MC-LSTM: Real-Time 3D Human Action Detection System for Intelligent Healthcare Applications10.1109/TBCAS.2021.3064841 ↗
30. Chen, Y.-T., Fang, W.-H., Dai, S.-T., and Lu, C.-C. (2021, January 24–25). Skeleton Moving Pose-Based Human Fall Detection with Sparse Coding and Temporal Pyramid Pooling. Proceedings of the 2021 7th International Conference on Applied System Innovation (ICASI), Chiayi, Taiwan.10.1109/ICASI52993.2021.9568464 ↗
31. Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.-M. (2020). CFAD: Coarse-to-Fine Action Detector for Spatiotemporal Action Localization. Computer Vision—ECCV 2020, Springer International Publishing.
32. Wang, Q., Zhang, Y., Zheng, Y., and Pan, P. (2022, January 19–24). RCL: Recurrent Continuous Localization for Temporal Action Detection. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01320 ↗
33. Jiang, B., Zhang, Z., Lin, D., Tang, J., and Luo, B. (2019, January 15–20). Semi-Supervised Learning With Graph Learning-Convolutional Networks. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01157 ↗
34. Shi, L., Zhang, Y., Cheng, J., and Lu, H. (2019, January 15–20). Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01230 ↗
35. Cheng, K., Zhang, Y., He, X., Chen, W., Cheng, J., and Lu, H. (2020, January 13–19). Skeleton-Based Action Recognition With Shift Graph Convolutional Network. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00026 ↗
36. Martin, M., Voit, M., and Stiefelhagen, R. (2020, January 20–23). Dynamic Interaction Graphs for Driver Activity Recognition. Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece.10.1109/ITSC45102.2020.9294520 ↗
37. Caetano, C., Sena, J., Brémond, F., Dos Santos, J.A., and Schwartz, W.R. (2019, January 18–21). SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition. Proceedings of the 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Taipei, Taiwan.10.1109/AVSS.2019.8909840 ↗
38. Le, T.M., Inoue, N., and Shinoda, K. (2018). A Fine-to-Coarse Convolutional Neural Network for 3D Human Action Recognition. arXiv.
39. Topology-Aware Convolutional Neural Network for Efficient Skeleton-Based Action Recognition
40. Lee, I., Kim, D., Kang, S., and Lee, S. (2017, January 22–29). Ensemble Deep Learning for Skeleton-Based Action Recognition Using Temporal Sliding LSTM Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.115 ↗
41. Liu, J., Wang, G., Hu, P., Duan, L.-Y., and Kot, A.C. (2017, January 21–26). Global Context-Aware Attention LSTM Networks for 3D Action Recognition. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.391 ↗
42. Environmental Factors-Aware Two-Stream GCN for Skeleton-Based Behavior Recognition10.1007/s00138-024-01656-7 ↗
43. SparseShift-GCN: High Precision Skeleton-Based Action Recognition10.1016/j.patrec.2021.12.005 ↗
44. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
45. Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks10.1109/TIP.2020.3028207 ↗
46. Abdelfattah, M., Hassan, M., and Alahi, A. (2024, January 17–21). MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01767 ↗
47. Advancing Skeleton-Based Human Behavior Recognition: Multi-Stream Fusion Spatiotemporal Graph Convolutional Networks10.1007/s40747-024-01743-2 ↗
48. Wang, W., Xie, W., Tu, Z., Li, W., and Jin, L. (2022, January 18–23). Multi-Part Adaptive Graph Convolutional Network for Skeleton-Based Action Recognition. Proceedings of the 2022 International Joint Conference on Neural Networks (IJCNN), Padua, Italy.10.1109/IJCNN55064.2022.9892727 ↗
49. SpatioTemporal Focus for Skeleton-Based Action Recognition10.1016/j.patcog.2022.109231 ↗
50. Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., and Tian, Q. (2019, January 15–20). Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00371 ↗
51. Peng, Z., Liu, H., Jia, Y., and Hou, J. (2021, January 20–24). Attention-Driven Graph Clustering Network. Proceedings of the 29th ACM International Conference on Multimedia, ACM Conferences, Chengdu, China.10.1145/3474085.3475276 ↗
52. Chi, H.-G., Ha, M.H., Chi, S., Lee, S.W., Huang, Q., and Ramani, K. (2022, January 19–24). InfoGCN: Representation Learning for Human Skeleton-Based Action Recognition. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01955 ↗
53. Lee, J., Lee, M., Lee, D., and Lee, S. (2023, January 2–6). Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00958 ↗
54. Chen, C., and Chai, L. (2024, January 25–27). Multi-Attention Graph Convolutional Network for Skeleton-Based Action Recognition. Proceedings of the 2024 36th Chinese Control and Decision Conference (CCDC), Xi′an, China.10.1109/CCDC62350.2024.10587517 ↗
55. Recognition of Miner Action and Violation Behavior Based on the ANODE-GCN Model10.1007/s00530-024-01547-x ↗
56. Skeleton-Based Violation Action Recognition Method for Safety Supervision in Operation Field of Distribution Network Based on Graph Convolutional Network
57. Li, P., Lu, M., Zhang, Z., Shan, D., and Yang, Y. (2019, January 27–30). A Novel Spatial-Temporal Graph for Skeleton-Based Driver Action Recognition. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.10.1109/ITSC.2019.8916929 ↗
58. Lin, Z., Liu, Y., and Zhang, X. (2021, January 19–22). Driver-Skeleton: A Dataset for Driver Action Recognition. Proceedings of the 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA.10.1109/ITSC48978.2021.9564922 ↗
59. An Effective Multi-Scale Framework for Driver Behavior Recognition With Incomplete Skeletons10.1109/TVT.2023.3308566 ↗
60. Lightweight Multimodal Feature Graph Convolutional Network for Dangerous Driving Behavior Detection10.1007/s11554-023-01277-9 ↗
61. Multi-Scale Spatial–Temporal Convolutional Neural Network for Skeleton-Based Action Recognition10.1007/s10044-023-01156-w ↗
62. Chao, Y.-W., Vijayanarasimhan, S., Seybold, B., Ross, D.A., Deng, J., and Sukthankar, R. (2018, January 18–22). Rethinking the Faster R-CNN Architecture for Temporal Action Localization. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00124 ↗
63. Boundary Graph Convolutional Network for Temporal Action Detection10.1016/j.imavis.2021.104144 ↗
64. A Temporal-Aware Relation and Attention Network for Temporal Action Localization10.1109/TIP.2022.3182866 ↗
65. Liu, X., Bai, S., and Bai, X. (2022, January 19–24). An Empirical Study of End-to-End Temporal Action Detection. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01938 ↗
66. Overview of Temporal Action Detection Based on Deep Learning10.1007/s10462-023-10650-w ↗
67. Sooksatra, S., and Watcharapinchai, S. (2022). A Comprehensive Review on Temporal-Action Proposal Generation. J. Imaging, 8.10.3390/jimaging8080207 ↗
68. Application of Temporal Action Detection Technology in Abnormal Event Detection of Surveillance Video10.1109/ACCESS.2025.3533552 ↗
69. Action Progression Networks for Temporal Action Detection in Videos10.1109/ACCESS.2024.3451503 ↗