How to Correctly Detect Face-Masks for COVID-19 from Visual Information?
Source abstract
Abstract
The new Coronavirus disease (COVID-19) has seriously affected the world. By the end of November 2020, the global number of new coronavirus cases had already exceeded 60 million and the number of deaths 1,410,378 according to information from the World Health Organization (WHO). To limit the spread of the disease, mandatory face-mask rules are now becoming common in public settings around the world. Additionally, many public service providers require customers to wear face-masks in accordance with predefined rules (e.g., covering both mouth and nose) when using public services. These developments inspired research into automatic (computer-vision-based) techniques for face-mask detection that can help monitor public behavior and contribute towards constraining the COVID-19 pandemic. Although existing research in this area resulted in efficient techniques for face-mask detection, these usually operate under the assumption that modern face detectors provide perfect detection performance (even for masked faces) and that the main goal of the techniques is to detect the presence of face-masks only. In this study, we revisit these common assumptions and explore the following research questions: (i) How well do existing face detectors perform with masked-face images? (ii) Is it possible to detect a proper (regulation-compliant) placement of facial masks? and iii) How useful are existing face-mask detection techniques for monitoring applications during the COVID-19 pandemic? To answer these and related questions we conduct a comprehensive experimental evaluation of several recent face detectors for their performance with masked-face images. Furthermore, we investigate the usefulness of multiple off-the-shelf deep-learning models for recognizing correct face-mask placement. Finally, we design a complete pipeline for recognizing whether face-masks are worn correctly or not and compare the performance of the pipeline with standard face-mask detection models from the literature. To facilitate the study, we compile a large dataset of facial images from the publicly available MAFA and Wider Face datasets and annotate it with compliant and non-compliant labels. The annotation dataset, called Face-Mask-Label Dataset (FMLD), is made publicly available to the research community.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face recognition and analysis · Video Surveillance and Tracking Methods · Face and Expression Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Computer and Information Science, University of Ljubljana, Večna Pot 113, SI-1000 Ljubljana, Slovenia
Faculty of Electrical Engineering, University of Ljubljana, Tržaška Cesta 25, SI-1000 Ljubljana, Slovenia
References (67)
Show source-provided references
1. World Health Organization (2020, December 01). Coronavirus Disease (COVID-19) Advice for the Public. Available online: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-for-public.
2. Centers for Disease Control and Prevention (2020, November 12). Considerations for Wearing Masks, Available online: https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/cloth-face-cover-guidance.html.
3. Respiratory virus shedding in exhaled breath and efficacy of face masks10.1038/s41591-020-0843-2 ↗
4. Rational use of face masks in the COVID-19 pandemic10.1016/S2213-2600(20)30134-X ↗
5. Wang, Z., Wang, G., Huang, B., Xiong, Z., Hong, Q., Wu, H., Yi, P., Jiang, K., Wang, N., and Pei, Y. (2020). Masked face recognition dataset and application. arXiv.
6. A survey on face detection in the wild: Past, present and future10.1016/j.cviu.2015.03.015 ↗
7. Wang, Y. (2020, November 26). Which Mask Are You Wearing? Face Mask Type Detection with TensorFlow and Raspberry Pi. Available online: https://towardsdatascience.com/which-mask-are-you-wearing-face-mask-type-detection-with-tensorflow-and-raspberry-pi-1c7004641f1.
8. Pranilshinde (2020, November 26). Face-Mask Detector Using TensorFlow-Object Detection (SSD_MobileNet). Available online: https://medium.com/pranil-shinde/face-mask-detector-using-tensorflow-object-detection-ssd-mobilenet-37f233202c67.
9. Bhandary, P. (2020, November 26). Mask Classifier. Available online: https://github.com/prajnasb/observations.
10. Rosebrock, A. (2020, November 26). COVID-19: Face Mask Detector with OpenCV, Keras/TensorFlow, and Deep Learning. Available online: https://www.pyimagesearch.com/2020/05/04/covid-19-face-mask-detector-with-opencv-keras-tensorflow-and-deep-learning.
11. Khanna, R. (2020, November 26). COVID-19-Authorized-Entry-Using-Face-Mask-Detection. Available online: https://github.com/Rahul24-06/COVID-19-Authorized-Entry-using-Face-Mask-Detection.
12. Bornstein, A.A. (2020, November 26). Personal Face Mask Detection with Custom Vision and Tensorflow.js. Available online: https://medium.com/microsoftazure/corona-face-mask-detection-with-custom-vision-and-tensorflow-js-86e5fff84373.
13. Yicong, O. (2020, November 26). Python Face Masks Detection Project. Available online: https://github.com/ohyicong/masksdetection.
14. K, G.M. (2020, November 26). COVID-19: Face Mask Detection Using TensorFlow and OpenCV. Available online: https://towardsdatascience.com/covid-19-face-mask-detection-using-tensorflow-and-opencv-702dd833515b.
15. Song, W. (2020, November 26). COVID19 Face Mask Detection Using Deep Learning. Available online: https://www.mathworks.com/matlabcentral/fileexchange/76758-covid19-face-mask-detection-using-deep-learning.
16. A hybrid deep transfer learning model with machine learning methodsfor face mask detection in the era of the COVID-19 pandemic10.1016/j.measurement.2020.108288 ↗
17. Chowdary, G.J., Punn, N.S., Sonbhadra, S.K., and Agarwal, S. (2020). Face mask detection using transfer learning of inceptionv3. Proceedings of the International Conference on Big Data Analytics, Sonepat, India, 15–18 December 2020, Springer.
18. Ge, S., Li, J., Ye, Q., and Luo, Z. (2017, January 21–26). Detecting masked faces in the wild with LLE-CNNs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.53 ↗
19. (2020, November 26). RetinaFace Anti Cov Face Detector. Available online: https://github.com/deepinsight/insightface/tree/master/detection/RetinaFaceAntiCov.
20. Daniell Chiang, Y.F. (2020, November 26). Detect Faces and Determine Whether they are Wearing Mask. Available online: https://github.com/AIZOOTech/FaceMaskDetection.
21. Jiang, M., and Fan, X. (2020). RetinaMask: A Face Mask detector. arXiv.
22. Yang, S., Luo, P., Loy, C.C., and Tang, X. (2016, January 27–30). Wider face: A face detection benchmark. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.596 ↗
23. Ngan, M.L., Grother, P.J., and Hanaoka, K.K. (2020). Ongoing face recognition vendor test (FRVT) Part 6A: Face recognition accuracy with masks using pre-COVID-19 algorithms. NIST Interagency/Internal Report (NISTIR), NIST.10.6028/NIST.IR.8331 ↗
24. Damer, N., Grebe, J.H., Chen, C., Boutros, F., Kirchbuchner, F., and Kuijper, A. (2020, January 16–18). The effect of wearing a mask on face recognition performance: An exploratory study. Proceedings of the 2020 International Conference of the Biometrics Special Interest Group (BIOSIG), Darmstadt, Germany.
25. Wang, J., Yuan, Y., and Yu, G. (2017). Face attention network: An effective face detector for the occluded faces. arXiv.
26. Larxel (2020, November 26). Face Mask Detection, 853 Images Belonging to 3 Classes. Available online: https://www.kaggle.com/andrewmvd/face-mask-detection.
27. Wobot Intelligence (2020, November 26). Face Mask Detection Dataset. Available online: https://www.kaggle.com/wobotintelligence/face-mask-detection-dataset.
28. Humans in the Loop (2020, November 26). Medical Mask Dataset. Available online: https://humansintheloop.org/medical-mask-dataset.
29. Deng, J., Guo, J., Ververas, E., Kotsia, I., and Zafeiriou, S. (2020, January 14–19). RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Conference on Computer Vision and Pattern Recognition (CVPR2020), Seattle, WA, USA (Virtual).10.1109/CVPR42600.2020.00525 ↗
30. Khandelwal, P., Khandelwal, A., and Agarwal, S. (2020). Using Computer Vision to enhance Safety of Workforce in Manufacturing in a Post COVID World. arXiv.
31. Qin, B., and Li, D. (2020). Identifying Facemask-wearing Condition Using Image Super-Resolution with Classification Network to Prevent COVID-19. Sensors, 20.10.21203/rs.3.rs-28668/v1 ↗
32. Trident (2020, November 26). Face Mask Detection System Using Artificial Intelligence. Available online: https://www.tridentinfo.com/face-mask-detection-systems.
33. Leewayhertz (2020, November 26). Face Mask Detection System Using Artificial Intelligence. Available online: https://www.leewayhertz.com/face-mask-detection-system.
34. Udemans, C. (2020, November 26). Baidu Releases Open-Source Tool to Detect Faces without Masks. Available online: https://technode.com/2020/02/14/baidu-open-source-face-masks.
35. Tang, X., Du, D.K., He, Z., and Liu, J. (2018, January 8–14). Pyramidbox: A context-assisted single shot face detector. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01240-3_49 ↗
36. Joint face detection and alignment using multitask cascaded convolutional networks10.1109/LSP.2016.2603342 ↗
37. Villán, A.F. (2019). Mastering OpenCV 4 with Python: A Practical Guide Covering Topics from Image Processing, Augmented Reality to Deep Learning with OpenCV 4 and Python 3.7, Packt Publishing Ltd.
38. Li, J., Wang, Y., Wang, C., Tai, Y., Qian, J., Yang, J., Wang, C., Li, J., and Huang, F. (2019, January 16–20). DSFD: Dual shot face detector. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00520 ↗
39. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 8–16). Ssd: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
40. The pascal visual object classes challenge: A retrospective10.1007/s11263-014-0733-5 ↗
41. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
42. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft coco: Common objects in context. Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
43. Cartucho, J., Ventura, R., and Veloso, M. (2018, January 1–5). Robust Object Recognition Through Symbiotic Deep Learning In Mobile Robots. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8594067 ↗
44. Deep learning10.1038/nature14539 ↗
45. Application of the residue number system to reduce hardware costs of the convolutional neural network implementation10.1016/j.matcom.2020.04.031 ↗
46. Deep content-based music recommendation
47. Collobert, R., and Weston, J. (2008, January 5–9). A unified architecture for natural language processing: Deep neural networks with multitask learning. Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland.10.1145/1390156.1390177 ↗
48. Subject-independent brain-computer interfaces based on deep convolutional neural networks10.1109/TNNLS.2019.2946869 ↗
49. Tsantekidis, A., Passalis, N., Tefas, A., Kanniainen, J., Gabbouj, M., and Iosifidis, A. (2017, January 24–27). Forecasting stock prices from the limit order book using convolutional neural networks. Proceedings of the 2017 IEEE 19th Conference on Business Informatics (CBI), Thessaloniki, Greece.10.1109/CBI.2017.23 ↗
50. Bortolato, B., Ivanovska, M., Rot, P., Križaj, J., Terhorst, P., Damer, N., Peer, P., and Štruc, V. (2020, January 16–20). Learning privacy-enhancing face representations through feature disentanglement. Proceedings of the FG 2020, IEEE International Conference on Automatic Face & Gesture Recognition, Buenos Aires, Argentina (Virtual).10.1109/FG47880.2020.00007 ↗
51. Jiang, R., Li, C., Crookes, D., Meng, W., and Rosenberger, C. (2020). Constellation-Based Deep Ear Recognition. Deep Biometrics: Unsupervised and Semi-Supervised Learning, Springer.10.1007/978-3-030-32583-1 ↗
52. Vitek, M., Rot, P., Štruc, V., and Peer, P. (2020). A comprehensive investigation into sclera biometrics: A novel dataset and performance study. Neural Comput. Appl., 1–15.10.1007/s00521-020-04782-1 ↗
53. Deep face recognition for surveillance applications
54. Krizhevsky, A. (2014). One weird trick for parallelizing convolutional neural networks. arXiv.
55. Strengths and weaknesses of deep learning models for face recognition against image degradations10.1049/iet-bmt.2017.0083 ↗
56. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
57. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
58. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size. arXiv.
59. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
60. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
61. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–23). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
62. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
63. Pinkney, J. (2020, November 26). MTCNN Face Detection. Available online: https://github.com/matlab-deep-learning/mtcnn-face-detection/releases/tag/v1.2.3.
64. Rosebrock, A. (2020, November 26). Face Detection with OpenCV and Deep Learning. Available online: https://www.pyimagesearch.com/2018/02/26/face-detection-with-opencv-and-deep-learning/.
65. Hukkelås, H. (2020, November 26). DSFD-Pytorch-Inference. Available online: https://github.com/hukkelas/DSFD-Pytorch-Inference.
66. biubug6 (2020, November 26). RetinaFace in PyTorch. Available online: https://github.com/biubug6/Pytorch_Retinaface.
67. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22–29). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.74 ↗