Generating Scenery Images with Larger Variety According to User Descriptions
Source abstract
Abstract
In this paper, a framework based on generative adversarial networks is proposed to perform nature-scenery generation according to descriptions from the users. The desired place, time and seasons of the generated scenes can be specified with the help of text-to-image generation techniques. The framework improves and modifies the architecture of a generative adversarial network with attention models by adding the imagination models. The proposed attentional and imaginative generative network uses the hidden layer information to initialize the memory cell of the recurrent neural network to produce the desired photos. A data set containing different categories of scenery images is established to train the proposed system. The experiments validate that the proposed method is able to increase the quality and diversity of the generated images compared to the existing method. A possible application of road image generation for data augmentation is also demonstrated in the experimental results.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Computer Graphics and Visualization Techniques · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Information Engineering, National Central University, 300 Jong-Da Rd., Jong-Li, Taoyuan 32001, Taiwan
Department of Information and Computer Engineering, Chung Yuan Christian University, 200 Chung Pei Road, Chung-Li, Taoyuan 32023, Taiwan
Funding
ministry of science and technology
References (40)
Show source-provided references
1. Ian, J.G., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Networksvol. Proceedings of the International Conference on Neural Information Processing Systems (NIPS), MIT Press.
2. Recent Progress on Generative Adversarial Networks (GANs): A Survey10.1109/ACCESS.2019.2905015 ↗
3. Radford, A., and Metz, L. (2016, January 2–4). Unsupervised representation learning with deep convolutional generative adversarial networks. Proceedings of the International Conference on Learning Representations, San Juan, Puerto Rico.
4. Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv, Available online: https://arxiv.org/abs/1411.1784.
5. Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (2016). InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. arXiv.
6. Odena, A., Olah, C., and Shlens, J. (2017, January 6–11). Conditional image synthesis with auxiliary classier GANs. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
7. Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6–11). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia.
8. Metz, L., Poole, B., Pfau, D., and Sohl-Dickstein, J. (2017, January 24–26). Unrolled generative adversarial networks. Proceedings of the Proceedings International Conference on Learning Representations, Toulon, France.
9. Nguyen, T.D., Le, T., Vu, H., and Phung, D. (2017, January 4–9). Dual Discriminator Generative Adversarial Nets. Proceedings of the Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA.
10. Chen, Q., and Koltun, V. (2017, January 22–29). Photographic image synthesis with cascaded refinement networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.168 ↗
11. Ozkan, S., and Ozkan, A. (2018, January 7–10). Kinshipgan: Synthesizing of Kinship Faces from Family Photos by Regularizing a Deep Face Network. Proceedings of the IEEE International Conference on Image Processing (ICIP), Athens, Greece.10.1109/ICIP.2018.8451305 ↗
12. Chen, W., and Hays, J. (2018, January 18–22). SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis. Proceedings of the IEEE Con-ference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00981 ↗
13. Chen, Y.S., Wang, Y.C., Kao, M.H., and Chuang, Y.Y. (2018, January 18–23). Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00660 ↗
14. Zhang, Z., Xie, Y., and Yang, L. (2018, January 18–23). Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00649 ↗
15. Gregor, K., Danihelka, I., Graves, A., Rezende, D.J., and Wierstra, D. (2015, January 7–9). DRAW: A Recurrent Neural Network for Image Generation. Proceedings of the International Conference on Machine Learning, Lille, France.
16. Jin, Y., Zhang, J., Li, M., Tian, Y., and Zhu, H. (2017, January 8). Towards the high-quality anime characters generation with generative adversarial network. Proceedings of the Machine Learning for Creativity and Design, NIPS Workshop, Long Beach, CA, USA.
17. van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., and Kavukcuoglu, K. (2016, January 5–10). Conditional Image Generation with PixelCNN Decoders. Proceedings of the Conference on Neural Information Processing Systems, Barcelona, Spain.
18. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2017, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
19. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2017, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
20. Globally and locally consistent image completion10.1145/3072959.3073659 ↗
21. Generative adversarial network: An overview of theory and applications
22. Point Encoder GAN: A deep learning model for 3D point cloud inpainting10.1016/j.neucom.2019.12.032 ↗
23. Shin, H., Tenenholtz, N.A., Tenenholtz, A., Rogers, J., Schwarz, C., Senjem, M., Gunter, J., Andriole, K., and Michalski, M. (2018, January 16). Medical Image Synthesis for Data Augmentation and Anonymization Using Generative Adversarial Networks. Proceedings of the International Workshop on Simulation and Synthesis in Medical Imaging, Granada, Spain.10.1007/978-3-030-00536-8_1 ↗
24. Tian, Y., Pei, K., Jana, S., and Ray, B. (June, January 27). DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars. Proceedings of the International Conference on Software Engineering 2018, Gothenburg, Sweden.10.1145/3180155.3180220 ↗
25. Fang, H., Gupta, S., Iandola, F.N., Srivastava, R.K., Deng, L., Doll´ar, P., Gao, J., He, X., Mitchell, M., and Platt, J.C. (2015, January 7–12). From captions to visual concepts and back. Proceedings of the IEEE Conference on Computer Vision and Pat-tern Recognition 2015, Boston, MA, USA.10.1109/CVPR.2015.7298754 ↗
26. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. (2016, January 19–24). Generative Adversarial Text to Image Synthesis. Proceedings of the International Conference on Machine Learning, New York City, NY, USA.
27. Gan, Z., Gan, C., He, X., Pu, Y., Tran, K., Gao, J., Carin, L., and Deng, L. (2017, January 21–26). Semantic compositional networks for visual captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2017, Honolulu, HI, USA.10.1109/CVPR.2017.127 ↗
28. Reed, S., Akata, Z., Schiele, B., and Lee, H. (2016, January 27–30). Learning Deep Representations of Fine-grained Visual Descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2016, Las Vegas, NV, USA.10.1109/CVPR.2016.13 ↗
29. Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., and Metaxas, D. (2017, January 22–29). StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.629 ↗
30. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. (2018, January 18–22). AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00143 ↗
31. Zhu, Y., Elhoseiny, M., Liu, B., Peng, X., and Elgammal, A. (2018, January 18–22). A generative adversarial approach for zero-shot learning from noisy texts. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00111 ↗
32. Yu, L., Zhang, W., Wang, J., and Yu, Y. (2017, January 15–18). SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), Montréal, QC, Canada.10.1609/aaai.v31i1.10804 ↗
33. Bidirectional Recurrent Neural Networks10.1109/78.650093 ↗
34. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale10.1007/s11263-020-01316-z ↗
35. A 10 million Image Database for Scene Recognition10.1109/TPAMI.2017.2723009 ↗
36. Improved Techniques for Training GANs
37. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017, January 4–9). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Proceedings of the International Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA.
38. Ying, G., Zou, Y., Wan, L., Hu, Y., and Feng, J. (2018). Better guider predicts future better: Difference guided generative adversarial networks. Proceedings of the Asian Conference on Computer Vision, Perth, Australia, 2–6 December 2018, Springer.10.1007/978-3-030-20876-9_18 ↗
39. Animating landscape: Self-supervised learning of decoupled motion and appearance for single-image video synthesis10.1145/3355089.3356523 ↗
40. Kwon, Y., and Park, M.G. (2019, January 16–20). Predicting future frames using retrospective cycle GAN. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00191 ↗