Text-Guided Customizable Image Synthesis and Manipulation
Source abstract
Abstract
Due to the high flexibility and conformity to people’s usage habits, text description has been widely used in image synthesis research recently and has achieved many encouraging results. However, the text can only determine the basic content of the generated image and cannot determine the specific shape of the synthesized object, which leads to poor practicability. More importantly, the current text-to-image synthesis research cannot use new text descriptions to further modify the synthesis result. To solve these problems, this paper proposes a text-guided customizable image synthesis and manipulation method. The proposed method synthesizes the corresponding image based on the text and contour information at first. It then modifies the synthesized content based on the new text to obtain a satisfactory result. The text and contour information in the proposed method determine the specific content and object shape of the desired composite image, respectively. Aside from that, the input text, contour, and subsequent new text for content modification can be manually input, which significantly improves the artificial controllability in the image synthesis process, making the entire method superior to other methods in flexibility and practicability. Experimental results on the Caltech-UCSD Birds-200-2011 (CUB) and Microsoft Common Objects in Context (MS COCO) datasets demonstrate our proposed method’s feasibility and versatility.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Advanced Image and Video Retrieval Techniques · Image Processing and 3D Reconstruction
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Science and Engineering, Hosei University, Tokyo 184-8584, Japan
Institute of Liberal Arts and Science, Kanazawa University, Kanazawa 920-1192, Japan
Funding
Joint Research Project of Young Research of Hosei University
References (35)
Show source-provided references
1. Kingma, D.P., and Welling, M. (2014, January 14–16). Auto-Encoding Variational Bayes. Proceedings of the International Conference on Learning Representation, Banff, AB, Canada.
2. Van Den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016, January 19–24). Pixel Recurrent Neural Networks. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
3. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., and Bengio, Y. (2014, January 8–13). Generative Adversarial Nets. Proceedings of the Advances in Neural Information Processing Systems 27, Montreal, QC, Canada.
4. Mirza, M., and Osindero, S. (2014). Conditional Generative Adversarial Nets. arXiv.
5. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. (2018, January 18–22). AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. Proceedings of the Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00143 ↗
6. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The Caltech-UCSD Birds-200-2011 Dataset. CNS-TR-2011-001, California Institute of Technology.
7. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
8. Radford, A., Metz, L., and Chintala, S. (2016, January 2–4). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico.
9. Salimans, T., Goodfellow, I.J., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016, January 5–10). Improved Techniques for Training GANs. Proceedings of the Advances in Neural Information Processing Systems 29, Barcelona, Spain.
10. Reed, S.E., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. (2016, January 19–24). Generative Adversarial Text to Image Synthesis. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
11. Zhang, H., Xu, T., and Li, H. (2017, January 22–29). StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks. Proceedings of the International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.629 ↗
12. Zhao, B., Meng, L., Yin, W., and Sigal, L. (2019, January 16–20). Image Generation From Layout. Proceedings of the Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00878 ↗
13. Gao, C., Liu, Q., Xu, Q., Wang, L., Liu, J., and Zou, C. (2020, January 13–19). SketchyCOCO: Image Generation From Freehand Scene Sketches. Proceedings of the Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00522 ↗
14. Wang, S.-Y., Bau, D., and Zhu, J.-Y. (2021, January 10–17). Sketch Your Own GAN. Proceedings of the International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01379 ↗
15. Reed, S.E., Akata, Z., Mohan, S., Tenka, S., Schiele, B., and Lee, H. (2016, January 5–10). Learning What and Where to Draw. Proceedings of the Advances in Neural Information Processing Systems 29, Barcelona, Spain.
16. Customizable GAN: A Method for Image Synthesis of Human Controllable10.1109/ACCESS.2020.3001070 ↗
17. Brock, A., Lim, T., Ritchie, J.M., and Weston, N. (2017, January 24–26). Neural Photo Editing with Introspective Adversarial Networks. Proceedings of the International Conference on Learning Representations, Toulon, France.
18. Liu, M.-Y., Breuel, T.M., and Kautz, J. (2017, January 4–9). Unsupervised Image-to-Image Translation Networks. Proceedings of the Advances in Neural Information Processing Systems 30, Long Beach, CA, USA.
19. Dong, H., Yu, S., Wu, C., and Guo, Y. (2017, January 22-29). Semantic Image Synthesis via Adversarial Learning. Proceedings of the International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.608 ↗
20. Nam, S., Kim, Y., and Kim, S.J. (2018, January 3–8). Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language. Proceedings of the Advances in Neural Information Processing Systems 31, Montréal, QC, Canada.
21. Li, B., Qi, X., Lukasiewicz, T., and Torr, P.H.S. (2020, January 13–19). ManiGAN: Text-Guided Image Manipulation. Proceedings of the Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00790 ↗
22. Che, T., Li, Y., Zhang, R., Hjelm, R.D., Li, W., Song, Y., and Bengio, Y. (2017). Maximum-Likelihood Augmented Discrete Generative Adversarial Networks. arXiv.
23. Vondrick, C., Pirsiavash, H., and Torralba, A. (2016, January 5–10). Generating Videos with Scene Dynamics. Proceedings of the Advances in Neural Information Processing Systems 29, Barcelona, Spain.
24. Bidirectional recurrent neural networks10.1109/78.650093 ↗
25. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the Inception Architecture for Computer Vision. Proceedings of the Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
26. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
27. Ulyanov, D., Vedaldi, A., and Lempitsky, V.S. (2016). Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv.
28. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the International Conference on Machine Learning, Lille, France.
29. Li, B., Qi, X., Lukasiewicz, T., and Torr, P.H.S. (2019, January 8–14). Controllable Text-to-Image Generation. Proceedings of the Advances in Neural Information Processing Systems 32, Vancouver, BC, Canada.
30. Zhu, M., Pan, P., Chen, W., and Yang, Y. (2019, January 16–20). DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis. Proceedings of the Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00595 ↗
31. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
32. Johnson, J., Alahi, A., and Fei-Fei, L. (2016, January 11–14). Perceptual Losses for Real-Time Style Transfer and Super-Resolution. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46475-6_43 ↗
33. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
34. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017, January 4–9). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Proceedings of the Advances in Neural Information Processing Systems 30, Long Beach, CA, USA.
35. StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks10.1109/TPAMI.2018.2856256 ↗