Holistic Interpretation of Public Scenes Using Computer Vision and Temporal Graphs to Identify Social Distancing Violations
Source abstract
Abstract
Social distancing measures are proposed as the primary strategy to curb the spread of the COVID-19 pandemic. Therefore, identifying situations where these protocols are violated has implications for curtailing the spread of the disease and promoting a sustainable lifestyle. This paper proposes a novel computer vision-based system to analyze CCTV footage to provide a threat level assessment of COVID-19 spread. The system strives to holistically interpret the information in CCTV footage spanning multiple frames to recognize instances of various violations of social distancing protocols, across time and space, as well as identification of group behaviors. This functionality is achieved primarily by utilizing a temporal graph-based structure to represent the information of the CCTV footage and a strategy to holistically interpret the graph and quantify the threat level of the given scene. The individual components are evaluated in a range of scenarios, and the complete system is tested against human expert opinion. The results reflect the dependence of the threat level on people, their physical proximity, interactions, protective clothing, and group dynamics, with a system performance of 76% accuracy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Digital Mental Health Interventions · COVID-19 epidemiological studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical and Electronic Engineering, University of Peradeniya, Peradeniya 20400, Sri Lanka
Department of Computer Science, University of Maryland, College Park, MD 20742, USA
Department of Informatics, Technical University of Munich, 85748 Garching, Germany
School of Engineering, Cardiff University, Cardiff CF24 3AA, Wales, UK
Department of Community Medicine, University of Peradeniya, Peradeniya 20400, Sri Lanka
Funding
International Development Research Centre (IDRC) · Lewis Power, Singapore
References (61)
Show source-provided references
1. A comparative study on the clinical features of coronavirus 2019 (COVID-19) pneumonia with other pneumonias10.1093/cid/ciaa247 ↗
2. Cardiovascular complications in COVID-1910.1016/j.ajem.2020.04.048 ↗
3. Neurological associations of COVID-1910.1016/S1474-4422(20)30221-0 ↗
4. Effectiveness of Covid-19 Vaccines against the B.1.617.2 (Delta) Variant10.1056/NEJMoa2108891 ↗
5. SARS-CoV-2 immune evasion by variant B. 1.427/B. 1.42910.1126/science.abi7994 ↗
6. COVID-19 vaccine efficacy and effectiveness—The elephant (not) in the room10.1016/S2666-5247(21)00069-0 ↗
7. Pormohammad, A., Zarei, M., Ghorbani, S., Mohammadi, M., Razizadeh, M.H., Turner, D.L., and Turner, R.J. (2021). Efficacy and Safety of COVID-19 Vaccines: A Systematic Review and Meta-Analysis of Randomized Clinical Trials. Vaccines, 9.10.2139/ssrn.3812422 ↗
8. Abdullahi, L., Onyango, J.J., Mukiira, C., Wamicwe, J., Githiomi, R., Kariuki, D., Mugambi, C., Wanjohi, P., Githuka, G., and Nzioka, C. (2020). Community interventions in Low—And Middle-Income Countries to inform COVID-19 control implementation decisions in Kenya: A rapid systematic review. PLoS ONE, 15.10.1371/journal.pone.0242403 ↗
9. Mukerjee, S., Chow, C.M., and Li, M. (2021). Mitigation strategies and compliance in the COVID-19 fight; how much compliance is enough?. PLoS ONE, 16.10.1371/journal.pone.0239352 ↗
10. Exploring the nexus between social vulnerability, built environment, and the prevalence of COVID-19: A case study of Chicago10.1016/j.scs.2021.103261 ↗
11. Social distancing enhanced automated optimal design of physical spaces in the wake of the COVID-19 pandemic10.1016/j.scs.2021.102791 ↗
12. Mitigating COVID-19 infection disease transmission in indoor environment using physical barriers10.1016/j.scs.2021.103175 ↗
13. Effective ventilation and air disinfection system for reducing Coronavirus Disease 2019 (COVID-19) infection risk in office buildings10.1016/j.scs.2021.103408 ↗
14. Digital contact tracing, community uptake, and proximity awareness technology to fight COVID-19: A systematic review10.1016/j.scs.2021.102995 ↗
15. Bian, S., Zhou, B., Bello, H., and Lukowicz, P. (2020, January 12–17). A wearable magnetic field based proximity sensing system for monitoring COVID-19 social distancing. Proceedings of the 2020 International Symposium on Wearable Computers, Virtual Event.10.1145/3410531.3414313 ↗
16. Fazio, M., Buzachis, A., Galletta, A., Celesti, A., and Villari, M. (2020, January 7–10). A proximity-based indoor navigation system tackling the COVID-19 social distancing measures. Proceedings of the IEEE Symposium on Computers and Communications, Rennes, France.10.1109/ISCC50000.2020.9219634 ↗
17. Shafiee, M.J., Chywl, B., Li, F., and Wong, A. (2017). Fast YOLO: A fast you only look once system for real-time embedded object detection in video. arXiv.10.15353/vsnl.v3i1.171 ↗
18. Wojke, N., Bewley, A., and Paulus, D. (2017, January 17–20). Simple online and realtime tracking with a deep association metric. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.10.1109/ICIP.2017.8296962 ↗
19. Automatic License Plate Recognition via sliding-window darknet-YOLO deep learning10.1016/j.imavis.2019.04.007 ↗
20. A two-stage real-time YOLOv2-based road marking detector with lightweight spatial transformation-invariant classification10.1016/j.imavis.2020.103978 ↗
21. Efficient pedestrian detection in top-view fisheye images using compositions of perspective view patches10.1016/j.imavis.2020.104069 ↗
22. Using channel pruning-based YOLO v4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments10.1016/j.compag.2020.105742 ↗
23. Monitoring social distancing through human detection for preventing/reducing COVID spread
24. Social distance monitoring framework using deep learning architecture to control infection transmission of COVID-19 pandemic10.1016/j.scs.2021.102777 ↗
25. A deep learning-based social distance monitoring framework for COVID-1910.1016/j.scs.2020.102571 ↗
26. Reaserch and implementation of social distancing monitoring technology based on SSD10.1016/j.procs.2021.02.127 ↗
27. Rahim, A., Maqbool, A., and Rana, T. (2021). Monitoring social distancing under various low light conditions with deep learning and a single motionless time of flight camera. PLoS ONE, 16.10.1371/journal.pone.0247440 ↗
28. A novel social distancing analysis in urban public space: A new online spatio-temporal trajectory approach10.1016/j.scs.2021.102765 ↗
29. Rezaei, M., and Azarmi, M. (2020). Deepsocial: Social distancing monitoring and infection risk assessment in covid-19 pandemic. Appl. Sci., 10.10.21203/rs.3.rs-68650/v1 ↗
30. Yang, D., Yurtsever, E., Renganathan, V., Redmill, K.A., and Özgüner, Ü. (2021). A vision-based social distancing and critical density detection system for COVID-19. Sensors, 21.10.3390/s21134608 ↗
31. Punn, N.S., Sonbhadra, S.K., and Agarwal, S. (2020). Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques. arXiv.
32. To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic
33. Potential role of inanimate surfaces for the spread of coronaviruses and their inactivation with disinfectant agents10.1016/j.infpip.2020.100044 ↗
34. Human coronavirus 229E remains infectious on common touch surface materials10.1128/mBio.01697-15 ↗
35. Moxa: A deep learning based unmanned approach for real-time monitoring of people wearing medical masks10.1007/s41403-020-00157-z ↗
36. Mohan, P., Paul, A.J., and Chirania, A. (2021). A tiny CNN architecture for medical face mask detection for resource-constrained endpoints. Innovations in Electrical and Electronic Engineering, Springer.10.1007/978-981-16-0749-3_52 ↗
37. Fighting against COVID-19: A novel deep learning model based on YOLO-v2 with ResNet-50 for medical face mask detection10.1016/j.scs.2020.102600 ↗
38. FMD-Yolo: An efficient face mask detection method for COVID-19 prevention and control in public10.1016/j.imavis.2021.104341 ↗
39. Hassan, A., Sritharan, S., Jayatilaka, G., Godaliyadda, R.I., Ekanayake, P.B., Herath, V., and Ekanayake, J.B. (2021). Hands Off: A Handshake Interaction Detection and Localization Model for COVID-19 Threat Control. arXiv.
40. YOLO based human action recognition and localization10.1016/j.procs.2018.07.112 ↗
41. Discriminative key-component models for interaction detection and recognition10.1016/j.cviu.2015.02.012 ↗
42. Hands-on: Deformable pose and motion models for spatiotemporal localization of fine-grained dyadic interactions10.1186/s13640-018-0255-0 ↗
43. Two metres or one: What is the evidence for physical distancing in COVID-19?10.1136/bmj.m3223 ↗
44. Association of social distancing and face mask use with risk of COVID-1910.1038/s41467-021-24115-7 ↗
45. Touch in times of COVID-19: Touch hunger hurts10.1111/jocn.15488 ↗
46. COVID-19 and social distancing10.1007/s10389-020-01321-z ↗
47. Social distancing policies in 22 African countries during the COVID-19 pandemic: A desk review10.11604/pamj.supp.2020.37.1.27026 ↗
48. Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv.
49. A computational approach to edge detection10.1109/TPAMI.1986.4767851 ↗
50. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014). Microsoft coco: Common objects in context. Proceedings of the European Conference on Computer Vision, Springer.10.1007/978-3-319-10602-1_48 ↗
51. Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., and Leal-Taixé, L. (2020). Mot20: A benchmark for multi object tracking in crowded scenes. arXiv.
52. Jiang, Y., Gao, F., and Xu, G. (2010, January 9–11). Computer vision-based multiple-lane detection on straight road and in a curve. Proceedings of the 2010 International Conference on Image Analysis and Signal Processing, Xiamen, China.
53. A tutorial on spectral clustering10.1007/s11222-007-9033-z ↗
54. Milan, A., Leal-Taixé, L., Reid, I., Roth, S., and Schindler, K. (2016). MOT16: A benchmark for multi-object tracking. arXiv.
55. Leal-Taixé, L., Milan, A., Reid, I., Roth, S., and Schindler, K. (2015). Motchallenge 2015: Towards a benchmark for multi-target tracking. arXiv.
56. Ryoo, M.S., and Aggarwal, J.K. (2022, August 14). UT-Interaction Dataset, ICPR contest on Semantic Description of Human Activities (SDHA). Available online: http://cvrc.ece.utexas.edu/SDHA2010/Human_Interaction.html.
57. Ryoo, M.S., and Aggarwal, J.K. (October, January 29). Spatio-Temporal Relationship Match: Video Structure Comparison for Recognition of Complex Human Activities. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Kyoto, Japan.
58. The pascal visual object classes (voc) challenge10.1007/s11263-009-0275-4 ↗
59. Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., and Kolesnikov, A. (2018). The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv.
60. Robertson, S. (2008, January 20–24). A new interpretation of average precision. Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Singapore.10.1145/1390334.1390453 ↗
61. Padilla, R., Netto, S.L., and da Silva, E.A.B. (2020, January 1–3). A Survey on Performance Metrics for Object-Detection Algorithms. Proceedings of the 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), Niteroi, Brazil.10.1109/IWSSIP48289.2020.9145130 ↗