Thermal Image Generation for Robust Face Recognition
Source abstract
Abstract
This article shows how to create a robust thermal face recognition system based on the FaceNet architecture. We propose a method for generating thermal images to create a thermal face database with six different attributes (frown, glasses, rotation, normal, vocal, and smile) based on various deep learning models. First, we use StyleCLIP, which oversees manipulating the latent space of the input visible image to add the desired attributes to the visible face. Second, we use the GANs N’ Roses (GNR) model, a multimodal image-to-image framework. It uses maps of style and content to generate thermal imaging from visible images, using generative adversarial approaches. Using the proposed generator system, we create a database of synthetic thermal faces composed of more than 100k images corresponding to 3227 individuals. When trained and tested using the synthetic database, the Thermal-FaceNet model obtained a 99.98% accuracy. Furthermore, when tested with a real database, the accuracy was more than 98%, validating the proposed thermal images generator system.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face recognition and analysis · Generative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Escuela de Ingeniería Eléctrica, Pontificia Universidad Católica de Valparaíso, Av. Brasil 2147, Valparaíso 2362804, Chile
Funding
PUCV
References (59)
Show source-provided references
1. Highly accurate protein structure prediction with AlphaFold10.1038/s41586-021-03819-2 ↗
2. Mastering Atari Games with Limited Data
3. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., and Graepel, T. (2019). Mastering atari, go, chess and shogi by planning with a learned model. arXiv.10.1038/s41586-020-03051-4 ↗
4. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., and Sutskever, I. (2021). Zero-Shot Text-to-Image Generation. arXiv.
5. Knowledge elicitation based on genetic programming for non destructive testing of critical aerospace systems10.1016/j.future.2019.09.007 ↗
6. Szegedy, C., Ioffe, S., and Vanhoucke, V. (2017, January 4–9). Inception-v4, Inception-Resnet and the Impact of Residual Connections on Learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
7. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv.
8. Schroff, F., Kalenichenko, D., and Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. arXiv.10.1109/CVPR.2015.7298682 ↗
9. Deng, J., Guo, J., and Zafeiriou, S. (2018). ArcFace: Additive angular margin loss for deep face recognition. arXiv.10.1109/CVPR.2019.00482 ↗
10. Socolinsky, D., and Selinger, A. (2002, January 11–15). A Comparative Analysis of Face Recognition Performance with Visible and Thermal Infrared Imagery. Proceedings of the International Conference on Pattern Recognition (ICPR), Quebec City, QC, Canada.10.21236/ADA453159 ↗
11. Selinger, A., and Socolinsky, D.A. (2001). Appearance-Based Facial Recognition Using Visible and Thermal Imagery: A Comparative Study, Equinox Corporation.
12. Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv.
13. Doersch, C. (2016). Tutorial on variational autoencoders. arXiv.
14. Diederik, P.K., and Max, W. (2019). An Introduction to Variational Autoencoders. arXiv.
15. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative Adversarial Networks (PDF). Proceedings of the International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
16. Karras, T., Laine, S., and Aila, T. (2019, January 15–20). A Style-Based Generator Architecture for Generative Adversarial Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00453 ↗
17. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2019). Analyzing and improving the image quality of stylegan. arXiv.10.1109/CVPR42600.2020.00813 ↗
18. Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. (2020). Training generative adversarial networks with limited data. arXiv.
19. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015, January 6–11). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France.
20. HO, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. arXiv.
21. Nichol, A., and Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. arXiv.
22. Dhariwal, P., and Nichol, A. (2021). Diffusion Models Beat GANs on Image Synthesis. arXiv.
23. Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. (2021). Styleclip: Text-driven manipulation of stylegan imagery. arXiv.10.1109/ICCV48922.2021.00209 ↗
24. Chong, M.J., and Forsyth, D. (2021). GANs N’Roses: Stable, Controllable, Diverse Image to Image Translation (works for videos too). arXiv.
25. Brock, A., Donahue, J., and Simonyan, K. (2018). Large scale gan training for high fidelity natural image synthesis. arXiv.
26. Durall, R., Pfreundt, F.-J., and Keuper, J. (2020). Local facial attribute transfer through inpainting. arXiv.
27. Laxman, K., Dubey, S.R., Kalyan, B., and Kojjarapu, S.R.V. (2021). Efficient High-Resolution Image-to-Image Translation using Multi-Scale Gradient U-Net. arXiv.10.1007/978-3-031-11346-8_4 ↗
28. Jam, J., Kendrick, C., Drouard, V., Walker, K., Hsu, G.-S., and Yap, M.H. (2021, January 5–9). R-mnet: A Perceptual Adversarial Network for Image Inpainting. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikola, HI, USA.10.1109/WACV48630.2021.00276 ↗
29. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (2018). Free-form image inpainting with gated convolution. arXiv.10.1109/ICCV.2019.00457 ↗
30. Khan, K., Mauro, M., and Leonardi, R. (2015, January 27–30). Multi-Class Semantic Segmentation of Faces. Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada.10.1109/ICIP.2015.7350915 ↗
31. Kalayeh, M.M., Gong, B., and Shah, M. (2017, January 21–26). Improving Facial Attribute Prediction using Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.450 ↗
32. Child, R. (2020). Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images. arXiv.
33. Esser, P., Rombach, R., and Ommer, B. (2020). Taming Transformers for High-Resolution Image Synthesis. arXiv.10.1109/CVPR46437.2021.01268 ↗
34. Nie, W., Karras, T., Garg, A., Debhath, S., Patney, A., Patel, A.B., and Anandkumar, A. (2020, January 12–18). Semisupervised Stylegan for Disentanglement Learning. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria.
35. Shen, Y., Yang, C., Tang, X., and Zhou, B. (2020). Interfacegan: Interpreting the disentangled face representation learned by gans. arXiv.
36. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning transferable visual models from natural language supervision. arXiv.
37. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language models are few-shot learners. arXiv.
38. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
39. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2015). Rethinking the inception architecture for computer vision. arXiv.10.1109/CVPR.2016.308 ↗
40. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2018, January 18–23). Mobilenetv2: Inverted Residuals and Linear Bottlenecks. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
41. Huang, G., Liu, Z., Weinberger, K.Q., and Maaten, L. (2017, January 21–26). Densely Connected Convolutional Networks. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
42. Chollet, F. (2016). Xception: Deep learning with depthwise separable convolutions. arXiv.10.1109/CVPR.2017.195 ↗
43. Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., and Song, L. (2017, January 21–26). Sphereface: Deep Hypersphere Embedding for Face Recognition. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.713 ↗
44. Cao, Q., Shen, L., Xie, W., Parkhi, O.M., and Zisserman, A. (2018, January 15–19). Vggface2: A Dataset for Recognising Faces across Pose and Age. Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG), Xi’an, China.10.1109/FG.2018.00020 ↗
45. Zhong, Y., and Deng, W. (2021). Face Transformer for Recognition. arXiv.
46. Fusion of visible and thermal descriptors using genetic algorithms for face recognition systems10.3390/s150817944 ↗
47. Joint face detection and alignment using multi-task cascaded convolutional networks10.1109/LSP.2016.2603342 ↗
48. Li, X., Chen, C., Zhou, S., Lin, X., Zuo, W., and Zhang, L. (2020, January 23–28). Blind Face Restoration via Deep Multi-Scale Component Dictionaries. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58545-7_23 ↗
49. Huang, X., and Belongie, S. (2017, January 22–29). Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. Proceedings of the International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.167 ↗
50. Mescheder, L., Geiger, A., and Nowozin, S. (2018). Which training methods for gans do actually converge?. arXiv.
51. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016, January 5–10). Improved Techniques for Training Gans. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.
52. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗
53. Isola, P., Zhu, P.J.-Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-Toimage Translation with Conditional Adversarial Networks. Proceedings of the Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
54. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
55. Li, C., and Wand, M. (2016, January 11–14). Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.10.1007/978-3-319-46487-9_43 ↗
56. Zeiler, M.D., and Fergus, R. (2014, January 6–12). Visualizing and Understanding Convolutional Networks. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10590-1_53 ↗
57. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going Deeper with Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
58. A comparative study of thermal face recognition methods in unconstrained environments10.1016/j.patcog.2012.01.001 ↗
59. Thermal Face Generation Using StyleGAN10.1109/ACCESS.2021.3085423 ↗