A 3D Occlusion Facial Recognition Network Based on a Multi-Feature Combination Threshold
Source abstract
Abstract
In this work, we propose a 3D occlusion facial recognition network based on a multi-feature combination threshold (MFCT-3DOFRNet). First, we design and extract the depth information of the 3D face point cloud, the elevation, and the azimuth angle of the normal vector as new 3D facially distinctive features, so as to improve the differentiation between 3D faces. Next, we propose a multi-feature combinatorial threshold that will be embedded at the input of the backbone network to implement the removal of occlusion features in each channel image. To enhance the feature extraction capability of the neural network for missing faces, we also introduce a missing face data generation method that enhances the training samples of the network. Finally, we use a Focal-ArcFace loss function to increase the inter-class decision boundaries and improve network performance during the training process. The experimental results show that the method has excellent recognition performance for unoccluded faces and also effectively improves the performance of 3D occlusion face recognition. The average Top-1 recognition rate of the proposed MFCT-3DOFRNet for the Bosphorus database is 99.52%, including 98.94% for occluded faces and 100% for unoccluded faces. For the UMB-DB dataset, the average Top-1 recognition rate is 95.08%, including 93.41% for occluded faces and 100% for unoccluded faces. These 3D face recognition experiments show that the proposed method essentially meets the requirements of high accuracy and good robustness.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face recognition and analysis · Face and Expression Recognition · Biometric Identification and Security
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China
University of Chinese Academy of Sciences, Beijing 100049, China
College of Electronic and Information Engineering, Suzhou University of Science and Technology, Suzhou 215009, China
Funding
the Opening Project of Key Laboratory of Sichuan Universities of Criminal Examination
References (68)
Show source-provided references
1. Face recognition: Challenges, achievements and future directions10.1049/iet-cvi.2014.0084 ↗
2. Structured Light from Lasers10.1002/lpor.201900140 ↗
3. State of the art and challenges of time-of-flight PET
4. Cheng, L., Chen, S., Liu, X., Xu, H., Wu, Y., Li, M., and Chen, Y. (2018). Registration of Laser Scanning Point Clouds: A Review. Sensors, 18.10.3390/s18051641 ↗
5. A Survey on the ICP Algorithm and Its Variants in Registration of 3D Point Clouds
6. Huang, X., Mei, G., Zhang, J., and Abbas, R. (2021). A comprehensive survey on point cloud registration. arXiv.
7. A comprehensive survey on 3D face recognition methods10.1016/j.engappai.2022.104669 ↗
8. 3D Face Recognition: A Survey
9. Occlusion detection and restoration techniques for 3D face recognition: A literature review10.1007/s00138-018-0933-z ↗
10. Gilani, S.Z., and Mian, A. (2018, January 18–23). IEEE In Learning from Millions of 3D Scans for Large-scale 3D Face Recognition. Proceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.
11. Ghazi, M.M., and Ekenel, H.K. (July, January 26). A Comprehensive Analysis of Deep Learning Based Representation for Face Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USA.10.1109/CVPRW.2016.20 ↗
12. A survey of face recognition techniques under occlusion10.1049/bme2.12029 ↗
13. Mathai, J., Masi, I., and AbdAlmageed, W. (2019, January 4–7). Does Generative Face Completion Help Face Recognition?. Proceedings of the 2019 International Conference on Biometrics (ICB), Crete, Greece.10.1109/ICB45273.2019.8987388 ↗
14. Occlusion Invariant 3D Face Recognition with UMB—Db and Bosporus Databases
15. 3D Face Recognition under Expressions, Occlusions, and Pose Variations10.1109/TPAMI.2013.48 ↗
16. 3D Face Recognition Using Geodesic Facial Curves to Handle Expression, Occlusion and Pose Variations
17. Yu, X., Gao, Y., and Zhou, J. (2016, January 25–28). 3D face recognition under partial occlusions using radial strings. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.10.1109/ICIP.2016.7532913 ↗
18. Yu, X., Gao, Y., and Zhou, J. (December, January 30). Boosting Radial Strings for 3D Face Recognition with Expressions and Occlusions. Proceedings of the 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Gold Coast, Australia.10.1109/DICTA.2016.7797014 ↗
19. Efficient 3D face recognition handling facial expression and hair occlusion10.1016/j.imavis.2012.07.011 ↗
20. Colombo, A., Cusano, C., and Schettini, R. (2006, January 9–12). Detection and Restoration of Occlusions for 3D Face Recognition. Proceedings of the 2006 IEEE International Conference on Multimedia and Expo, Toronto, ON, Canada.10.1109/ICME.2006.262837 ↗
21. Alyüz, N., Gökberk, B., Spreeuwers, L., Veldhuis, R., and Akarun, L. (April, January 29). Robust 3D face recognition in the presence of realistic occlusions. Proceedings of the 2012 5th IAPR International Conference on Biometrics (ICB), New Delhi, India.10.1109/ICB.2012.6199767 ↗
22. Bagchi, P., Bhattacharjee, D., and Nasipuri, M. (2014). Robust 3D face recognition in presence of pose and partial occlusions or missing parts. arXiv.10.5121/ijfcst.2014.4402 ↗
23. Alyuz, N., Gokberk, B., and Akarun, L. (2014, January 24–28). Detection of Realistic Facial Occlusions for Robust 3D Face Recognition. Proceedings of the 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden.10.1109/ICPR.2014.73 ↗
24. Zohra, F.T., Rahman, M.W., and Gavrilova, M. (2016, January 28–30). Occlusion Detection and Localization from Kinect Depth Images. Proceedings of the 2016 International Conference on Cyberworlds (CW), Chongqing, China.10.1109/CW.2016.40 ↗
25. Gappy wavelet neural network for 3D occluded faces: Detection and recognition10.1007/s11042-014-2294-6 ↗
26. Dutta, K., Bhattacharjee, D., and Nasipuri, M. (2016, January 23–24). Expression and occlusion invariant 3D face recognition based on region classifier. Proceedings of the 2016 1st International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE), Yogyakarta, Indonesia.10.1109/ICITISEE.2016.7803055 ↗
27. 3D geometry-based face recognition in presence of eye and mouth occlusions10.1007/s12008-019-00582-7 ↗
28. Accurate landmarking of three-dimensional facial data in the presence of facial expressions and occlusions using a three-dimensional statistical facial feature model10.1109/TSMCB.2011.2148711 ↗
29. Liu, R., Hu, R., and Yu, H. (2014, January 14–16). Nose detection on 3D face images by depth-based template matching. Proceedings of the 2014 7th International Congress on Image and Signal Processing, Dalian, China.10.1109/CISP.2014.7003796 ↗
30. Liu, P., Wang, Y., Huang, D., and Zhang, Z. (2012, January 9–13). Recognizing Occluded 3D Faces Using an Efficient ICP Variant. Proceedings of the 2012 IEEE International Conference on Multimedia and Expo, Melbourne, Australia.10.1109/ICME.2012.158 ↗
31. 3-D Face Recognition Under Occlusion Using Masked Projection10.1109/TIFS.2013.2256130 ↗
32. Gappy PCA Classification for Occlusion Tolerant 3D Face Detection10.1007/s10851-009-0165-y ↗
33. A survey on deep learning based face recognition10.1016/j.cviu.2019.102805 ↗
34. Mu, G., Huang, D., Hu, G., Sun, J., and Wang, Y. (2019, January 15–20). Led3D: A Lightweight and Efficient Deep Approach to Recognizing Low-Quality 3D Faces. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00592 ↗
35. PointFace: Point Cloud Encoder-Based Feature Embedding for 3-D Face Recognition10.1109/TBIOM.2022.3197437 ↗
36. Jan, A., Ding, H., Meng, H., Chen, L., and Li, H. (2018, January 15–19). Accurate Facial Parts Localization and Deep Learning for 3D Facial Expression Recognition. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China.10.1109/FG.2018.00075 ↗
37. Deep learning in optical metrology: A review
38. A modular hierarchical array camera10.1038/s41377-021-00485-x ↗
39. LOEN: Lensless opto-electronic neural network empowered machine vision10.1038/s41377-022-00809-5 ↗
40. Invalid point removal method based on error energy function in fringe projection profilometry10.1016/j.rinp.2022.105904 ↗
41. Deep 3D Facial Landmark Localization on position maps10.1016/j.neucom.2020.04.025 ↗
42. Survey on the approaches based geometric information for 3D face landmarks detection10.1049/iet-ipr.2018.6117 ↗
43. Face Frontalization Using an Appearance-Flow-Based Convolutional Neural Network10.1109/TIP.2018.2883554 ↗
44. Least-Squares Fitting of Two 3-D Point Sets10.1109/TPAMI.1987.4767965 ↗
45. Multimodal 2D+3D Facial Expression Recognition with Deep Fusion Convolutional Neural Network10.1109/TMM.2017.2713408 ↗
46. Li, H., Huang, D., Chen, L., Wang, Y., and Morvan, J.M. (2012, January 23–27). A group of facial normal descriptors for recognizing 3D identical twins. Proceedings of the 2012 IEEE Fifth International Conference on Biometrics: Theory, Applications and Systems (BTAS), Arlington, VA, USA.10.1109/BTAS.2012.6374588 ↗
47. Li, H., Sun, J., and Chen, L. (2017, January 1–4). Location-sensitive sparse representation of deep normal patterns for expression-robust 3D face recognition. Proceedings of the 2017 IEEE International Joint Conference on Biometrics (IJCB), Denver, CO, USA.10.1109/BTAS.2017.8272703 ↗
48. Heckbert, P.S. (1994). Graphics Gems, Academic Press.
49. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H.J. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.
50. ArcFace: Additive Angular Margin Loss for Deep Face Recognition10.1109/TPAMI.2021.3087709 ↗
51. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal Loss for Dense Object Detection. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.324 ↗
52. He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., and Li, M. (2019, January 15–20). Bag of Tricks for Image Classification with Convolutional Neural Networks. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00065 ↗
53. Schouten, B., Juul, N.C., Drygajlo, A., and Tistarelli, M. (2008). Bosphorus Database for 3D Face Analysis, Springer. Biometrics and Identity Management, Berlin, Heidelberg, 2008.10.1007/978-3-540-89991-4 ↗
54. Comparative evaluation of 3D vs. 2D modality for automatic detection of facial action units10.1016/j.patcog.2011.07.022 ↗
55. Colombo, A., Cusano, C., and Schettini, R. (2011, January 6–13). UMB-DB: A database of partially occluded 3D faces. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.10.1109/ICCVW.2011.6130509 ↗
56. Križaj, J., Dobrišek, S., and Štruc, V. (2022). Making the Most of Single Sensor Information: A Novel Fusion Approach for 3D Face Recognition Using Region Covariance Descriptors and Gaussian Mixture Models. Sensors, 22.10.3390/s22062388 ↗
57. Satapathy, S.C., Biswal, B.N., Udgata, S.K., and Mandal, J.K. Range Face Image Registration Using ERFI from 3D Images. Proceedings of the 3rd International Conference on Frontiers of Intelligent Computing: Theory and Applications (FICTA) 2014.
58. Depth based Occlusion Detection and Localization from 3D Face Image10.5815/ijigsp.2015.05.03 ↗
59. Huang, G., Liu, Z., Maaten, L.V.D., and Weinberger, K.Q. (2017, January 21–26). Densely Connected Convolutional Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
60. Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., and Pang, R. (November, January 27). Searching for MobileNetV3. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00140 ↗
61. Tan, M., and Le, Q.V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. International conference on machine learning. arXiv.
62. Schroff, F., Kalenichenko, D., and Philbin, J. (2015, January 7–12). FaceNet: A unified embedding for face recognition and clustering. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298682 ↗
63. Zhou, J., Wang, Y., Sun, Z., Jia, Z., Feng, J., Shan, S., Ubul, K., and Guo, Z. (2018). MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices, Springer International Publishing. Biometric Recognition, Cham, 2018.
64. Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., and Song, L. (2017, January 21–26). SphereFace: Deep Hypersphere Embedding for Face Recognition. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.713 ↗
65. Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., Li, Z., and Liu, W. (2018, January 18–23). CosFace: Large Margin Cosine Loss for Deep Face Recognition. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00552 ↗
66. Zhu, C.C., Wan, X.T., Xie, S.R., Li, X.Q., and Gu, Y.Z. (2022, January 18–24). IEEE Computer Society. Occlusion-robust Face Alignment using A Viewpoint-invariant Hierarchical Network Architecture. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01083 ↗
67. Wang, Q., Qian, W.-Z., Lei, H., and Chen, L. (2023). Siamese Neural Pointnet: 3D Face Verification under Pose Interference and Partial Occlusion. Electronics, 12.10.3390/electronics12030620 ↗
68. Masked face recognition with convolutional visual self-attention network10.1016/j.neucom.2022.10.025 ↗