Unlocking Efficiency in Fine-Grained Compositional Image Synthesis: A Single-Generator Approach
Source abstract
Abstract
The use of Generative Adversarial Networks (GANs) has led to significant advancements in the field of compositional image synthesis. In particular, recent progress has focused on achieving synthesis at the semantic part level. However, to enhance performance at this level, existing approaches in the literature tend to prioritize performance over efficiency, utilizing separate local generators for each semantic part. This approach leads to a linear increase in the number of local generators, posing a fundamental challenge for large-scale compositional image synthesis at the semantic part level. In this paper, we introduce a novel model called Single-Generator Semantic-Style GAN (SSSGAN) to improve efficiency in this context. SSSGAN utilizes a single generator to synthesize all semantic parts, thereby reducing the required number of local generators to a constant value. Our experiments demonstrate that SSSGAN achieves superior efficiency while maintaining a minimal impact on performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Image Processing and 3D Reconstruction · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer & Information Science, Southwest University, Chongqing 400715, China
Funding
National Natural Science Foundation of China
References (72)
Show source-provided references
1. Introducing the Discrete Path Transform (DPT) and Its Applications in Signal Analysis, Artefact Removal and Spoken Word Recognition10.1016/j.dsp.2021.103158 ↗
2. Guariglia, E., and Silvestrov, S. (2016). Engineering Mathematics II, Springer.10.1007/978-3-319-42105-6 ↗
3. Hyperspectral image classification using wavelet transform-based smooth ordering
4. Guariglia, E. (2018). Harmonic Sierpinski Gasket and Applications. Entropy, 20.10.3390/e20090714 ↗
5. Guariglia, E. (2019). Primality, Fractality and Image Analysis. Entropy, 21.10.3390/e21030304 ↗
6. A Framework of Adaptive Multiscale Wavelet Decomposition for Signals on Undirected Graphs10.1109/TSP.2019.2896246 ↗
7. On the Weierstrass-Mandelbrot fractal function
8. On the Adequacy of Prototype Theory as a Theory of Concepts10.1016/0010-0277(81)90013-5 ↗
9. Building Machines That Learn and Think Like People10.1017/S0140525X16001837 ↗
10. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020, January 13–19). Analyzing and Improving the Image Quality of StyleGAN. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00813 ↗
11. Park, T., Liu, M.Y., Wang, T.C., and Zhu, J.Y. (2019, January 15–20). Semantic Image Synthesis With Spatially-Adaptive Normalization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00244 ↗
12. Shi, Y., Yang, X., Wan, Y., and Shen, X. (2022, January 18–24). SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01097 ↗
13. Representation Learning: A Review and New Perspectives10.1109/TPAMI.2013.50 ↗
14. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017, January 24–26). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France.
15. Kim, H., and Mnih, A. (2018, January 10–15). Disentangling by Factorising. Proceedings of the International Conference on Machine Learning (ICML), Stockholm, Sweden.
16. Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (2016, January 5–10). InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Barcelona, Spain.
17. Lin, Z., Thekumparampil, K., Fanti, G., and Oh, S. (2020, January 13–18). InfoGAN-CR and ModelCentrality: Self-supervised Model Training and Selection for Disentangling GANs. Proceedings of the International Conference on Machine Learning (ICML) 2020, Virtual.
18. Kazemi, H., Iranmanesh, S.M., and Nasrabadi, N. (2019, January 7–11). Style and Content Disentanglement in Generative Adversarial Networks. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV.2019.00095 ↗
19. Tran, L., Yin, X., and Liu, X. (2017, January 21–26). Disentangled Representation Learning GAN for Pose-Invariant Face Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.141 ↗
20. A Unified Feature Disentangler for Multi-Domain Image Translation and Manipulation
21. MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation
22. Explicit Disentanglement of Appearance and Perspective in Generative Models
23. Lorenz, D., Bereska, L., Milbich, T., and Ommer, B. (2019, January 16–20). Unsupervised Part-Based Disentangling of Object Shape and Appearance. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01121 ↗
24. Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry10.1109/TPAMI.2020.3013905 ↗
25. Liu, L., Jiang, X., Saerbeck, M., and Dauwels, J. (2022). EAD-GAN: A Generative Adversarial Network for Disentangling Affine Transforms in Images. IEEE Trans. Neural Netw. Learn. Syst., 1–11.10.1109/TNNLS.2022.3195533 ↗
26. Tewari, A., R, M.B., Pan, X., Fried, O., Agrawala, M., and Theobalt, C. (2022, January 18–24). Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance From Monocular Images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00157 ↗
27. Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C., and Yang, Y.L. (2019, January 15–20). HoloGAN: Unsupervised Learning of 3D Representations from Natural Images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/ICCV.2019.00768 ↗
28. Ladder Variational Autoencoders
29. Zhao, S., Song, J., and Ermon, S. (2017, January 6–11). Learning Hierarchical Features from Deep Generative Models. Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia.
30. Li, Z., Murkute, J.V., Gyawali, P.K., and Wang, L. (2020, January 26–30). Progressive Learning and Disentanglement of Hierarchical Representations. Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia.
31. Kaneko, T., Hiramatsu, K., and Kashino, K. (2018, January 18–23). Generative Adversarial Image Synthesis with Decision Tree Latent Controller. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00691 ↗
32. Alharbi, Y., and Wonka, P. (2020, January 19–24). Disentangled Image Generation through Structured Noise Injection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR42600.2020.00518 ↗
33. Inductive biases for deep learning of higher-level cognition
34. Arandjelović, R., and Zisserman, A. (2019). Object Discovery with a Copy-Pasting GAN. arXiv.
35. Compositional GAN: Learning Image-Conditional Binary Composition10.1007/s11263-020-01336-9 ↗
36. Sbai, O., Couprie, C., and Aubry, M. (2021, January 20–25). Surprising Image Compositions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPRW53098.2021.00438 ↗
37. Burgess, C.P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A. (2019). MONet: Unsupervised Scene Decomposition and Representation. arXiv.
38. Greff, K., Kaufman, R.L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L., Botvinick, M., and Lerchner, A. (2019, January 10–15). Multi-Object Representation Learning with Iterative Variational Inference. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA.
39. Liao, Y., Schwarz, K., Mescheder, L., and Geiger, A. (2020, January 13–19). Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00591 ↗
40. BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
41. Niemeyer, M., and Geiger, A. (2021, January 20–25). GIRAFFE: Representing Scenes As Compositional Generative Neural Feature Fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01129 ↗
42. Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views
43. Anciukevicius, T., Lampert, C.H., and Henderson, P. (2020). Object-Centric Image Generation with Factored Depths, Locations and Appearances. arXiv.
44. Unsupervised Object-Centric Video Generation and Decomposition in 3D
45. Singh, K.K., Ojha, U., and Lee, Y.J. (2019, January 15–20). FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00665 ↗
46. GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
47. Kwak, H., and Zhang, B.T. (2016). Generating Images Part by Part with Composite Generative Adversarial Networks. arXiv.
48. Chen, W., and Hays, J. (2018, January 18–22). SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00981 ↗
49. Lu, Y., Wu, S., Tai, Y.W., and Tang, C.K. (2018, January 18–22). Image Generation from Sketch Constraint Using Contextual GAN. Proceedings of the European Conference on Computer Vision (ECCV), Salt Lake City, UT, USA.10.1007/978-3-030-01270-0_13 ↗
50. Generating Photographic Faces From the Sketch Guided by Attribute Using GAN10.1109/ACCESS.2019.2899466 ↗
51. Ghosh, A., Zhang, R., Dokania, P.K., Wang, O., Efros, A.A., Torr, P.H.S., and Shechtman, E. (November, January 27). Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
52. DeepFaceDrawing: Deep Generation of Face Images from Sketches10.1145/3386569.3392386 ↗
53. Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., and Cohen-Or, D. (2021, January 19–25). Encoding in Style: A StyleGAN Encoder for Image-to-Image Translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00232 ↗
54. Wang, S.Y., Bau, D., and Zhu, J.Y. (2021, January 11–17). Sketch Your Own GAN. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Montreal, BC, Canada.10.1109/ICCV48922.2021.01379 ↗
55. Chen, Q., and Koltun, V. (2017, January 22–29). Photographic Image Synthesis with Cascaded Refinement Networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.168 ↗
56. Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
57. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-To-Image Translation With Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
58. Tang, H., Xu, D., Sebe, N., Wang, Y., Corso, J.J., and Yan, Y. (2019, January 16–20). Multi-Channel Attention Selection GAN With Cascaded Semantic Guidance for Cross-View Image Translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00252 ↗
59. Wang, Y., Qi, L., Chen, Y.C., Zhang, X., and Jia, J. (2021, January 11–17). Image Synthesis via Semantic Composition. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, Canada.10.1109/ICCV48922.2021.01349 ↗
60. Zhu, Z., Xu, Z., You, A., and Bai, X. (2020, January 14–19). Semantically Multi-Modal Image Synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00551 ↗
61. Zhu, P., Abdal, R., Qin, Y., and Wonka, P. (2020, January 14–19). SEAN: Image Synthesis With Semantic Region-Adaptive Normalization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00515 ↗
62. Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., and Catanzaro, B. (2018, January 18–23). High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00917 ↗
63. SofGAN: A Portrait Image Generator with Dynamic Styling
64. Thanh-Tung, H., and Tran, T. (2020). On Catastrophic Forgetting and Mode Collapse in Generative Adversarial Networks. arXiv.10.1109/IJCNN48605.2020.9207181 ↗
65. van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016, January 19–24). Pixel Recurrent Neural Networks. Proceedings of the International Conference on Machine Learning (ICML), New York City, NY, USA.
66. Karras, T., Laine, S., and Aila, T. (2019, January 16–20). A Style-Based Generator Architecture for Generative Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00453 ↗
67. Mescheder, L.M., Geiger, A., and Nowozin, S. (2018, January 10–15). Which Training Methods for GANs do Actually Converge?. Proceedings of the International Conference on Machine Learning (ICML), Stockholm, Sweden.
68. Lee, C., Liu, Z., Wu, L., and Luo, P. (2020, January 14–19). MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, Washington, USA.10.1109/CVPR42600.2020.00559 ↗
69. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
70. Improved Techniques for Training GANs
71. Visualizing data using t-SNE
72. Liu, Z., Luo, P., Qiu, S., Wang, X., and Tang, X. (2016, January 27–30). DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.124 ↗