A CNN-RNN Combined Structure for Real-World Violence Detection in Surveillance Cameras
Source abstract
Abstract
Surveillance cameras have been increasingly used in many public and private spaces in recent years to increase the security of those areas. Although many companies still recruit someone to monitor the cameras, the person recruited is more likely to miss some abnormal events in the camera feeds due to human error. Therefore, monitoring surveillance cameras could be a waste of time and energy. On the other hand, many researchers worked on surveillance data and proposed several methods to detect abnormal events automatically. As a result, if any anomalous happens in front of the surveillance cameras, it can be detected immediately. Therefore, we introduced a model for detecting abnormal events in the surveillance camera feed. In this work, we designed a model by implementing a well-known convolutional neural network (ResNet50) for extracting essential features of each frame of our input stream followed by a particular schema of recurrent neural networks (ConvLSTM) for detecting abnormal events in our time-series dataset. Furthermore, in contrast with previous works, which mainly focused on hand-crafted datasets, our dataset took real-time surveillance camera feeds with different subjects and environments. In addition, we classify normal and abnormal events and show the method’s ability to find the right category for each anomaly. Therefore, we categorized our data into three main and essential categories: the first groups mainly need firefighting service, while the second and third categories are about thefts and violent behaviour. We implemented the proposed method on the UCF-Crime dataset and achieved 81.71% in AUC, higher than other models like C3D on the same dataset. Our future work focuses on adding an attention layer to the existing model to detect more abnormal events.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Anomaly Detection Techniques and Applications · Network Security and Intrusion Detection · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Engineering and Applied Science, University of Regina, Regina, SK S4S 0A2, Canada
Funding
Natural Sciences and Engineering Research Council
References (65)
Show source-provided references
1. Video behaviour mining using a dynamic topic model10.1007/s11263-011-0510-7 ↗
2. Sulman, N., Sanocki, T., Goldgof, D., and Kasturi, R. (2008, January 8–11). How effective is human video surveillance performance?. Proceedings of the 2008 19th IEEE International Conference on Pattern Recognition, ICPR, Tampa, FL, USA.10.1109/ICPR.2008.4761655 ↗
3. Nguyen, T.N., and Meunier, J. (November, January 27). Anomaly detection in video sequence with appearance-motion correspondence. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, ICCV, Seoul, Korea.10.1109/ICCV.2019.00136 ↗
4. Hierarchical and networked vehicle surveillance in its: A survey10.1109/TITS.2016.2552778 ↗
5. Joint representation learning of appearance and motion for abnormal event detection10.1007/s00138-018-0961-8 ↗
6. Varadarajan, J., and Odobez, J.M. (October, January 27). Topic models for scene analysis and abnormality detection. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, Kyoto, Japan.10.1109/ICCVW.2009.5457456 ↗
7. A review of anomaly detection in automated surveillance10.1109/TSMCC.2012.2215319 ↗
8. Zweng, A., and Kampel, M. (2010, January 23–26). Unexpected human behavior recognition in image sequences using multiple features. Proceedings of the 2010 20th International Conference on Pattern Recognition, ICPR, Istanbul, Turkey.10.1109/ICPR.2010.98 ↗
9. Jodoin, P.M., Konrad, J., and Saligrama, V. (2008, January 9–11). Modeling background activity for behavior subtraction. Proceedings of the 2008 Second ACM/IEEE International Conference on Distributed Smart Cameras, Trento, Italy.10.1109/ICDSC.2008.4635683 ↗
10. Pointwise motion image (PMI): A novel motion representation and its applications to abnormality detection and behavior recognition10.1109/TCSVT.2009.2013503 ↗
11. Mecocci, A., Pannozzo, M., and Fumarola, A. (2003, January 31). Automatic detection of anomalous behavioural events for advanced real-time video surveillance. Proceedings of the 3rd International Workshop on Scientific Use of Submarine Cables and Related Technologies, Lugano, Switzerland.
12. Behavior modeling and abnormality detection based on semi-supervised learning method
13. Yao, B., Wang, L., and Zhu, S.C. (2008, January 23–28). Learning a scene contextual model for tracking and abnormality detection. Proceedings of the 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Anchorage, AK, USA.10.1109/CVPRW.2008.4563039 ↗
14. Sensor-based abnormal human-activity detection10.1109/TKDE.2007.1042 ↗
15. Benezeth, Y., Jodoin, P.M., Saligrama, V., and Rosenberger, C. (2009, January 20–25). Abnormal events detection based on spatio-temporal co-occurences. Proceedings of the 2009 IEEE conference on computer vision and pattern recognition CVPR, Miami, FL, USA.10.1109/CVPR.2009.5206686 ↗
16. Dong, N., Jia, Z., Shao, J., Xiong, Z., Li, Z., Liu, F., Zhao, J., and Peng, P. (September, January 29). Traffic abnormality detection through directional motion behavior map. Proceedings of the 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance, Boston, MA, USA.10.1109/AVSS.2010.61 ↗
17. Detecting and discriminating behavioural anomalies10.1016/j.patcog.2010.07.023 ↗
18. Zhang, J., and Liu, Z. (2008, January 20–23). Detecting abnormal motion of pedestrian in video. Proceedings of the 2008 International Conference on Information and Automation, Changsha, China.10.1109/ICINFA.2008.4607972 ↗
19. Ruff, L., Vandermeulen, R.A., Görnitz, N., Binder, A., Müller, E., Müller, K.R., and Kloft, M. (2019). Deep semi-supervised anomaly detection. arXiv.
20. Tang, Y.P., Wang, X.J., and Lu, H.F. (2009, January 24–26). Intelligent video analysis technology for elevator cage abnormality detection in computer vision. Proceedings of the 2009 Fourth International Conference on Computer Sciences and Convergence Information Technology, Seoul, Korea.10.1109/ICCIT.2009.206 ↗
21. Feng, J., Zhang, C., and Hao, P. (2010, January 23–26). Online learning with self-organizing maps for anomaly detection in crowd scenes. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.10.1109/ICPR.2010.878 ↗
22. Sharif, M.H., Uyaver, S., and Djeraba, C. (2010, January 5–7). Crowd behavior surveillance using Bhattacharyya distance metric. Proceedings of the International Symposium Computational Modeling of Objects Represented in Images, Buffalo, NY, USA.10.1007/978-3-642-12712-0_28 ↗
23. Xiang, G., and Min, W. (2010, January 15–17). Applying Semi-supervised cluster algorithm for anomaly detection. Proceedings of the 2010 Third International Symposium on Information Processing, Qingdao, China.10.1109/ISIP.2010.68 ↗
24. Wang, J., Neskovic, P., and Cooper, L.N. (2005, January 8–11). Pattern classification via single spheres. Proceedings of the 8th International Conference on Discovery Science, Singapore.10.1007/11563983_21 ↗
25. Liu, W., Luo, W., Lian, D., and Gao, S. (2018, January 18–23). Future frame prediction for anomaly detection–a new baseline. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Salt Lake City, UT, USA.10.1109/CVPR.2018.00684 ↗
26. Ergen, T., Mirza, A.H., and Kozat, S.S. (2017). Unsupervised and semi-supervised anomaly detection with LSTM neural networks. arXiv.
27. (2018, January 12). Available online: https://visionlab.uncc.edu/download/summary/60-data/477-ucf-anomaly-detection-dataset.
28. Toward supervised anomaly detection10.1613/jair.3623 ↗
29. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.510 ↗
30. Sultani, W., Chen, C., and Shah, M. (2018, January 18–23). Real-world anomaly detection in surveillance videos. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Salt Lake City, UT, USA.10.1109/CVPR.2018.00678 ↗
31. Chalapathy, R., and Chawla, S. (2019). Deep learning for anomaly detection: A survey. arXiv.10.1145/3394486.3406704 ↗
32. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
33. Gorokhov, O., Petrovskiy, M., and Mashechkin, I. (November, January 30). Convolutional neural networks for unsupervised anomaly detection in text data. Proceedings of the 18th International Conference on Intelligent Data Engineering and Automated Learning, Guilin, China.
34. Kim, Y. (2014). Convolutional neural networks for sentence classification. arXiv.10.3115/v1/D14-1181 ↗
35. Varga, D. (2020). Multi-pooled inception features for no-reference image quality assessment. Appl. Sci., 10.10.3390/app10062186 ↗
36. Kawahara, J., BenTaieb, A., and Hamarneh, G. (2016, January 13–16). Deep features to classify skin lesions. Proceedings of the 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), Prague, Czech Republic.10.1109/ISBI.2016.7493528 ↗
37. Deep-person: Learning discriminative deep features for person re-identification10.1016/j.patcog.2019.107036 ↗
38. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions10.1186/s40537-021-00444-8 ↗
39. Christiansen, P., Nielsen, L.N., Steen, K.A., Jørgensen, R.N., and Karstoft, H. (2016). DeepAnomaly: Combining background subtraction and deep learning for detecting obstacles and anomalies in an agricultural field. Sensors, 16.10.3390/s16111904 ↗
40. Dong, L., Zhang, Y., Wen, C., and Wu, H. (2016, January 16–18). Camera anomaly detection based on morphological analysis and deep learning. Proceedings of the 2016 IEEE International Conference on Digital Signal Processing (DSP), Beijing, China.10.1109/ICDSP.2016.7868559 ↗
41. Xu, D., Ricci, E., Yan, Y., Song, J., and Sebe, N. (2015). Learning deep representations of appearance and motion for anomalous event detection. arXiv.10.5244/C.29.8 ↗
42. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A.K., and Davis, L.S. (2016, January 27–30). Learning temporal regularity in video sequences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.86 ↗
43. Bayesian nonparametric approaches to abnormality detection in video surveillance10.1007/s40745-015-0030-3 ↗
44. Wang, H., and Schmid, C. (2013, January 1–8). Action recognition with improved trajectories. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.441 ↗
45. Motion interaction field for detection of abnormal interactions10.1007/s00138-016-0816-0 ↗
46. Advances in deep learning approaches for image tagging10.1017/ATSIP.2017.12 ↗
47. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
48. Ebrahimi Kahou, S., Michalski, V., Konda, K., Memisevic, R., and Pal, C. (2015, January 9–13). Recurrent neural networks for emotion recognition in video. Proceedings of the 2015 17th ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.10.1145/2818346.2830596 ↗
49. LSTM fully convolutional networks for time series classification10.1109/ACCESS.2017.2779939 ↗
50. Medel, J.R., and Savakis, A. (2016). Anomaly detection in video using predictive convolutional long short-term memory networks. arXiv.
51. Singh, A. (2017). Anomaly Detection for Temporal Data Using Long Short-Term Memory (LSTM). [Master’s Thesis, KTH Royal Institute of Technology].
52. Srivastava, N., Mansimov, E., and Salakhudinov, R. (2015, January 6–11). Unsupervised learning of video representations using lstms. Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France.
53. Detecting urban anomalies using multiple spatio-temporal data sources10.1145/3287079 ↗
54. Dong, Z., Qin, J., and Wang, Y. (2016, January 5–7). Multi-stream deep networks for person to person violence detection in videos. Proceedings of the 7th Chinese Conference on Pattern Recognition (CCPR), Chengdu, China.10.1007/978-981-10-3002-4_43 ↗
55. Spatial–temporal convolutional neural networks for anomaly detection and localization in crowded scenes10.1016/j.image.2016.06.007 ↗
56. Sudhakaran, S., and Lanz, O. (September, January 29). Learning to detect violent videos using convolutional long short-term memory. Proceedings of the 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Lecce, Italy.10.1109/AVSS.2017.8078468 ↗
57. Xingjian, S.H.I., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., and Woo, W.C. (2015, January 7–12). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
58. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
59. Alzubaidi, L., Fadhel, M.A., Al-Shamma, O., Zhang, J., Santamaría, J., Duan, Y., and R Oleiwi, S. (2020). Towards a better understanding of transfer learning for medical imaging: A case study. Appl. Sci., 10.10.3390/app10134523 ↗
60. Mahmood, A., Ospina, A.G., Bennamoun, M., An, S., Sohel, F., Boussaid, F., Hovey, R., Fisher, R.B., and Kendrick, G.A. (2020). Automatic hierarchical classification of kelps using deep residual features. Sensors, 20.10.3390/s20020447 ↗
61. Generalized low-rank approximation of matrices based on multiple transformation pairs10.1016/j.patcog.2020.107545 ↗
62. Zapata-Impata, B.S., Gil, P., and Torres, F. (2019). Learning spatio temporal tactile features with a ConvLSTM for the direction of slip detection. Sensors, 19.10.3390/s19030523 ↗
63. Moreira, D., Barandas, M., Rocha, T., Alves, P., Santos, R., Leonardo, R., Vieria, P., and Gamboa, H. (2021). Human Activity Recognition for Indoor Localization Using Smartphone Inertial Sensors. Sensors, 21.10.3390/s21186316 ↗
64. Zhong, J.X., Li, N., Kong, W., Liu, S., Li, T.H., and Li, G. (2019, January 15–20). Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00133 ↗
65. Lu, C., Shi, J., and Jia, J. (2013, January 1–8). Abnormal event detection at 150 fps in matlab. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.338 ↗