Evaluation of StyleGAN-CLIP Models in Text-to-Image Generation of Faces
Source abstract
Abstract
In this paper, we explore the generation of face images conditioned on a textual description, as well as the capabilities of the models in editing a machine-generated image on the basis of additional text prompts. We leverage open source state-of-the-art face image generators, StyleGAN models and couple these with the open source multimodal embedding space, CLIP, in an optimisation loop using the method in StyleCLIP to set up our experimental system. We make use of automatic metrics and human ratings to evaluate the results and, in addition, obtain insight into how much automatic metrics are correlated with human ratings. We found compelling evidence that both the text-to-image and editing models based on StyleGAN2 stand out as the better options. In addition, the automatic evaluation metrics are only weakly correlated with human ratings.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Face recognition and analysis · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Communications and Computer Engineering, University of Malta, MSD 2080 Msida, Malta
Institute of Linguistics and Language Technology, University of Malta, MSD 2080 Msida, Malta
Funding
Xjenza Malta (formerly Malta Council for Science and Technology) FUSION: Technology Development Programme
References (48)
Show source-provided references
1. A comprehensive survey and analysis of generative models in machine learning10.1016/j.cosrev.2020.100285 ↗
2. Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models10.1109/TPAMI.2021.3116668 ↗
3. A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications10.1109/TKDE.2021.3130191 ↗
4. Nguyen, N.T., Tran, T.K., Tukayev, U., Hong, T.P., Trawiński, B., and Szczerbicki, E. (2022). A Deep Convolution Generative Adversarial Network for the Production of Images of Human Faces. Intelligent Information and Database Systems, Proceedings of the 14th Asian Conference, ACIIDS 2022, Ho Chi Minh City, Vietnam, 28–30 November 2022, Springer.10.1007/978-3-031-21967-2 ↗
5. Han, X., Wu, Y., and Wan, R. (2023). A Method for Style Transfer from Artistic Images Based on Depth Extraction Generative Adversarial Network. Appl. Sci., 13.10.3390/app13020867 ↗
6. A Music-Driven Deep Generative Adversarial Model for Guzheng Playing Animation10.1109/TVCG.2021.3115902 ↗
7. Data augmentation using Generative Adversarial Networks (GANs) for GAN-based detection of Pneumonia and COVID-19 in chest X-ray images10.1016/j.imu.2021.100779 ↗
8. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative adversarial nets. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
9. Generative Adversarial Networks (GANs) in networking: A comprehensive survey and evaluation10.1016/j.comnet.2021.108149 ↗
10. Generative Adversarial Networks: An Overview10.1109/MSP.2017.2765202 ↗
11. Kim, M., Liu, F., Jain, A., and Liu, X. (2023). DCFace: Synthetic Face Generation with Dual Condition Diffusion Model. arXiv.10.1109/CVPR52729.2023.01223 ↗
12. Peng, Y., Zhao, C., Xie, H., Fukusato, T., and Miyata, K. (2023). DiffFaceSketch: High-Fidelity Face Image Synthesis with Sketch-Guided Latent Diffusion Model. arXiv.
13. Szeliga, A. (2023). A Comparative Study of Deep Generative Models for Image Generation. [Master’s Thesis, Hochschule Hannover].
14. Comparative Analysis of GANs and Diffusion Models in Image Generation10.54097/9gba9v27 ↗
15. Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv.
16. Ehrhart, M., Resch, B., Havas, C., and Niederseer, D. (2022). A Conditional GAN for Generating Time Series Data for Stress Detection in Wearable Physiological Sensor Data. Sensors, 22.10.3390/s22165969 ↗
17. Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (2016, January 5–10). InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. Proceedings of the NIPS’16: 30th International Conference on Neural Information Processing Systems, Barcelona, Spain.
18. Karras, T., Laine, S., and Aila, T. (2019, January 15–20). A Style-Based Generator Architecture for Generative Adversarial Networks. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00453 ↗
19. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020, January 13–19). Analyzing and Improving the Image Quality of StyleGAN. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00813 ↗
20. Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. (2021, January 6–14). Alias-Free Generative Adversarial Networks. Proceedings of the Neural Information Processing Systems, Online.
21. Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. (2021, January 10–17). StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00209 ↗
22. Sauer, A., Karras, T., Laine, S., Geiger, A., and Aila, T. (2023). StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis. arXiv.
23. CLIP-Guided StyleGAN Inversion for Text-Driven Real Image Editing10.1145/3610287 ↗
24. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the International Conference on Machine Learning, Online.
25. Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. (2020, January 6–12). Training Generative Adversarial Networks with Limited Data. Proceedings of the NIPS’20: 34th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
26. Viazovetskyi, Y., Ivashkin, V., and Kashin, E. (2020, January 23–28). StyleGAN2 Distillation for Feed-Forward Image Manipulation. Proceedings of the ECCV 2020, Glasgow, UK.10.1007/978-3-030-58542-6_11 ↗
27. Alaluf, Y., Patashnik, O., Wu, Z., Zamir, A., Shechtman, E., Lischinski, D., and Cohen-Or, D. (2022, January 23–27). Third Time’s the Charm? Image and Video Editing with StyleGAN3. Proceedings of the Computer Vision—ECCV 2022 Workshops, Tel Aviv, Israel.10.1007/978-3-031-25063-7_13 ↗
28. Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., and Langlotz, C. (2020, January 7–8). Contrastive Learning of Medical Visual Representations from Paired Images and Text. Proceedings of the Machine Learning in Health Care 2020, Online.
29. Herrera-Berg, E. (2024, March 21). StyleGAN3-CLIP-Notebooks. Available online: https://github.com/ouhenio/StyleGAN3-CLIP-notebooks.
30. GAN Inversion: A Survey10.1109/TPAMI.2022.3181070 ↗
31. Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.M. (2020). In-Domain GAN Inversion for Real Image Editing. Computer Vision—ECCV 2020, Proceedings of the 16th European Conference, Glasgow, UK, 23–28 August 2020, Springer.10.1007/978-3-030-58595-2 ↗
32. Shen, Y., Gu, J., Tang, X., and Zhou, B. (2020, January 13–19). Interpreting the Latent Space of GANs for Semantic Face Editing. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00926 ↗
33. Collins, E., Bala, R., Price, B., and Süsstrunk, S. (2020, January 13–19). Editing in Style: Uncovering the Local Semantics of GANs. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00581 ↗
34. Kang, M., Zhu, J.Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. (2023, January 17–24). Scaling up GANs for Text-to-Image Synthesis. Proceedings of the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00976 ↗
35. Pinkney, J.N.M., and Li, C. (2022, January 21–24). clip2latent: Text driven sampling of a pre-trained StyleGAN using denoising diffusion and CLIP. Proceedings of the British Machine Vision Conference 2022, London, UK.
36. Moens, M.F., Huang, X., Specia, L., and Yih, S.W.t. (2021, January 7–11). CLIPScore: A Reference-free Evaluation Metric for Image Captioning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.
37. Liu, X., Gong, C., Wu, L., Zhang, S., Su, H., and Liu, Q. (2021). FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimization. arXiv.
38. Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., and Hassner, T. (2022). TISE: Bag of Metrics for Text-to-Image Synthesis Evaluation. Computer Vision—ECCV 2022, Proceedings of the 17th European Conference, Tel Aviv, Israel, 23–27 October 2022, Springer.10.1007/978-3-031-19800-7 ↗
39. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. (2018, January 18–23). AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00143 ↗
40. Park, D.H., Azadi, S., Liu, X., Darrell, T., and Rohrbach, A. (2021, January 6–14). Benchmark for Compositional Text-to-Image Synthesis. Proceedings of the NeurIPS Datasets and Benchmarks 2021, Online.
41. Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., and Smith, N.A. (2023). TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. arXiv.10.1109/ICCV51070.2023.01866 ↗
42. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc.
43. Xia, X., Xu, C., and Nan, B. (2017, January 2–4). Inception-v3 for flower classification. Proceedings of the 2017 2nd International Conference on Image, Vision and Computing (ICIVC), Chengdu, China.
44. Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., and Kumar, S. (2024, January 16–22). Rethinking FID: Towards a Better Evaluation Metric for Image Generation. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00889 ↗
45. CLIP2GAN: Towards Bridging Text with the Latent Space of GANs10.1109/TCSVT.2023.3294261 ↗
46. Petsiuk, V. (2022). Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark. arXiv.
47. Otani, M., Togashi, R., Sawai, Y., Ishigami, R., Nakashima, Y., Rahtu, E., Heikkila, J., and Satoh, S. (2023, January 17–24). Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01372 ↗
48. Paggio, P., Gatt, A., and Tanti, M. (2022, January 20–25). Face2Text revisited: Improved data set and baseline results. Proceedings of the 2nd Workshop on People in Vision, Language, and the Mind, Marseille, France.