TextControlGAN: Text-to-Image Synthesis with Controllable Generative Adversarial Networks
Source abstract
Abstract
Generative adversarial networks (GANs) have demonstrated remarkable potential in the realm of text-to-image synthesis. Nevertheless, conventional GANs employing conditional latent space interpolation and manifold interpolation (GAN-CLS-INT) encounter challenges in generating images that accurately reflect the given text descriptions. To overcome these limitations, we introduce TextControlGAN, a controllable GAN-based model specifically designed for text-to-image synthesis tasks. In contrast to traditional GANs, TextControlGAN incorporates a neural network structure, known as a regressor, to effectively learn features from conditional texts. To further enhance the learning performance of the regressor, data augmentation techniques are employed. As a result, the generator within TextControlGAN can learn conditional texts more effectively, leading to the production of images that more closely adhere to the textual conditions. Furthermore, by concentrating the discriminator’s training efforts on GAN training exclusively, the overall quality of the generated images is significantly improved. Evaluations conducted on the Caltech-UCSD Birds-200 (CUB) dataset demonstrate that TextControlGAN surpasses the performance of the cGAN-based GAN-INT-CLS model, achieving a 17.6% improvement in Inception Score (IS) and a 36.6% reduction in Fréchet Inception Distance (FID). In supplementary experiments utilizing 128 × 128 resolution images, TextControlGAN exhibits a remarkable ability to manipulate minor features of the generated bird images according to the given text descriptions. These findings highlight the potential of TextControlGAN as a powerful tool for generating high-quality, text-conditioned images, paving the way for future advancements in the field of text-to-image synthesis.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Digital Media Forensic Detection · Computer Graphics and Visualization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electrical and Electronics Engineering, Chung-Ang University, Seoul 06974, Republic of Korea
Funding
National Research Foundation of Korea (NRF)
References (53)
Show source-provided references
1. Samek, W., Wiegand, T., and Müller, K.-R. (2017). Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv.
2. Lee, Y.-L., Tsung, P.-K., and Wu, M. (2018, January 16–19). Techology trend of edge ai. Proceedings of the 2018 International Symposium on VLSI Design, Automation and Test (VLSI-DAT), Hsinchu, Taiwan.10.1109/VLSI-DAT.2018.8373244 ↗
3. Ongsulee, P. (2017, January 22–24). Artificial intelligence, machine learning and deep learning. Proceedings of the 2017 15th International Conference on ICT and Knowledge Engineering (ICT&KE), Bangkok, Thailand.10.1109/ICTKE.2017.8259629 ↗
4. Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., and Frey, B. (2015). Adversarial autoencoders. arXiv.
5. Mescheder, L., Nowozin, S., and Geiger, A. (2017, January 6–11). Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
6. Generative adversarial networks in computer vision: A survey and taxonomy
7. Generative adversarial u-net for domain-free few-shot medical diagnosis10.1016/j.patrec.2022.03.022 ↗
8. Wang, F., Ma, Z., Zhang, X., Li, Q., and Wang, C. (2023). Ddsg-gan: Generative adversarial network with dual discriminators and single generator for black-box attacks. Mathematics, 11.10.3390/math11041016 ↗
9. Kim, M., and Song, M.H. (2023). High performing facial skin problem diagnosis with enhanced mask r-cnn and super resolution gan. Appl. Sci., 13.10.3390/app13020989 ↗
10. Wang, Y., and Zhang, S. (2023). Prediction of tumor lymph node metastasis using wasserstein distance-based generative adversarial networks combing with neural architecture search for predicting. Mathematics, 11.10.3390/math11030729 ↗
11. Generative adversarial networks10.1145/3422622 ↗
12. Hitawala, S. (2018). Comparative study on generative adversarial networks. arXiv.
13. Donahue, J., Krähenbühl, P., and Darrell, T. (2016). Adversarial feature learning. arXiv.
14. Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv.
15. A review on generative adversarial networks: Algorithms, theory, and applications10.1109/TKDE.2021.3130191 ↗
16. Generative adversarial network: An overview of theory and applications
17. Tulyakov, S., Liu, M.-Y., Yang, X., and Kautz, J. (2018, January 18–22). Mocogan: Decomposing motion and content for video generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00165 ↗
18. Zhu, J.-Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
19. Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. (2018, January 10–15). Image transformer. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
20. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021, January 18–24). Zero-shot text-to-image generation. Proceedings of the International Conference on Machine Learning, Virtual.
21. Icegan: Inverse covariance estimating generative adversarial network10.1088/2632-2153/acc638 ↗
22. Superstargan: Generative adversarial networks for image-to-image translation in large-scale domains10.1016/j.neunet.2023.02.042 ↗
23. Lee, M., and Seok, J. (2022). Score-guided generative adversarial networks. Axioms, 11.10.3390/axioms11120701 ↗
24. Inverse design of nanophotonic devices using generative adversarial networks10.1016/j.engappai.2022.105259 ↗
25. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. (2016, January 19–24). Generative adversarial text to image synthesis. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
26. Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., and Metaxas, D.N. (2017, January 22–29). Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.629 ↗
27. Stackgan++: Realistic image synthesis with stacked generative adversarial networks10.1109/TPAMI.2018.2856256 ↗
28. Mrp-gan: Multi-resolution parallel generative adversarial networks for text-to-image synthesis10.1016/j.patrec.2021.02.020 ↗
29. Text-to-image synthesis with self-supervised learning10.1016/j.patrec.2022.04.010 ↗
30. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A.C. (2017, January 4–9). Improved training of wasserstein gans. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
31. Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv.
32. Abnormal colon polyp image synthesis using conditional adversarial networks for improved detection performance10.1109/ACCESS.2018.2872717 ↗
33. Gauthier, J. (2014). Conditional Generative Adversarial Nets for Convolutional Face Generation, University of Stanford. Class Project for Stanford CS231N: Convolutional Neural Networks for Visual Recognition; Winter Semester.
34. Miyato, T., and Koyama, M. (2018). Cgans with projection discriminator. arXiv.
35. Odena, A., Olah, C., and Shlens, J. (2017, January 6–11). Conditional image synthesis with auxiliary classifier gans. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
36. Controllable generative adversarial network10.1109/ACCESS.2019.2899108 ↗
37. Perez, L., and Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv.
38. A survey on image data augmentation for deep learning10.1186/s40537-019-0197-0 ↗
39. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016, January 5–10). Improved techniques for training gans. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.
40. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The Caltech-UCSD Birds-200-2011 Dataset, California Institute of Technology.
41. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017, January 4–9). Gans trained by a two time-scale update rule converge to a local nash equilibrium. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
42. A new statistical-based kurtosis wavelet energy feature for texture recognition of sar images10.1109/TGRS.2012.2194787 ↗
43. Effective supervised multiple-feature learning for fused radar and optical data classification10.1049/iet-rsn.2016.0346 ↗
44. Combined method of an efficient cuckoo search algorithm and nonnegative matrix factorization of different zernike moment features for discrimination between oil spills and lookalikes in sar images10.1109/JSTARS.2018.2841503 ↗
45. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
46. Facial image synthesis and super-resolution with stacked generative adversarial network10.1016/j.neucom.2020.03.107 ↗
47. Yan, F., and Mikolajczyk, K. (2015, January 7–12). Deep correlation for matching images and text. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298966 ↗
48. Zero-shot cross-media embedding learning with dual adversarial distribution network10.1109/TCSVT.2019.2900171 ↗
49. Dash, A., Gamboa, J.C.B., Ahmed, S., Liwicki, M., and Afzal, M.Z. (2017). Tac-gan-text conditioned auxiliary classifier generative adversarial network. arXiv.
50. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
51. Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. arXiv.
52. Moore, R.C., and DeNero, J. (2011, January 21). L1 and l2 regularization for multiclass hinge loss models. Proceedings of the Symposium on Machine Learning in Speech and Natural Language Processing, Bellevue, WA, USA.
53. Ye, H., Yang, X., Takac, M., Sunderraman, R., and Ji, S. (2021). Improving text-to-image synthesis using contrastive learning. arXiv.