Editable Image Generation with Consistent Unsupervised Disentanglement Based on GAN
Source abstract
Abstract
Generative adversarial networks (GANs) are often used to generate realistic images, and GANs are effective in fitting high-dimensional probability distributions. However, during training, they often produce model collapse, which is the inability of the generative model to map the input noise to the real data distribution. In this work, we propose a model for disentanglement and mitigating model collapse inspired by the relationship between Hessian and Jacobian matrices. This is a concise framework for producing few modifications to the original model while facilitating the disentanglement. Compared to the pre-improvement generative models, our approach modifies the original model architecture only marginally and does not change the training method. Our method shows consistent resistance to model collapse on some image datasets, while outperforming the pre-improvement method in terms of disentanglement.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Digital Media Forensic Detection · Computer Graphics and Visualization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Anhui University of Science and Technology, Huainan 232001, China
Funding
The School Foundation of Anhui University of Science and Technology
References (46)
Show source-provided references
1. A Style-Based Generator Architecture for Generative Adversarial Networks10.1109/TPAMI.2020.2970919 ↗
2. Brock, A., Donahue, J., and Simonyan, K. (May, January 30). Large Scale GAN Training for High Fidelity Natural Image Synthesis. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
3. Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., and Choo, J. (2018, January 18–23). Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00916 ↗
4. Choi, Y., Uh, Y., Yoo, J., and Ha, J.W. (2020, January 13–19). Stargan v2: Diverse image synthesis for multiple domains. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00821 ↗
5. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020, January 13–19). Analyzing and improving the image quality of stylegan. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00813 ↗
6. Tolstikhin, I., Gelly, S., Bousquet, O., Simon-Gabriel, C.J., and Schölkopf, B. (2017, January 4–9). AdaGAN: Boosting generative models. Proceedings of the International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
7. Nguyen, T.D., Le, T., Vu, H., and Phung, D. (2017, January 4–9). Dual discriminator generative adversarial nets. Proceedings of the International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
8. Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6–11). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
9. Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., and Paul Smolley, S. (2017, January 21–26). Least squares generative adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Honolulu, HI, USA.10.1109/ICCV.2017.304 ↗
10. Zhao, J., Mathieu, M., and LeCun, Y. (2017, January 24–26). Energy-based generative adversarial networks. Proceedings of the International Conference on Learning Representations, Toulon, France.
11. Gurumurthy, S., Kiran Sarvadevabhatla, R., and Venkatesh Babu, R. (2017, January 21–26). Deligan: Generative adversarial networks for diverse and limited data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.525 ↗
12. Shen, Y., and Zhou, B. (2021, January 19–25). Closed-form factorization of latent semantics in gans. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Online.10.1109/CVPR46437.2021.00158 ↗
13. Peebles, W., Peebles, J., Zhu, J.Y., Efros, A., and Torralba, A. (2020, January 23–28). The hessian penalty: A weak prior for unsupervised disentanglement. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK.10.1007/978-3-030-58539-6_35 ↗
14. Wei, Y., Shi, Y., Liu, X., Ji, Z., Gao, Y., Wu, Z., and Zuo, W. (2021, January 19–25). Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Online.10.1109/ICCV48922.2021.00665 ↗
15. Denoising adversarial autoencoders10.1109/TNNLS.2018.2852738 ↗
16. Odena, A., Olah, C., and Shlens, J. (2017, January 6–11). Conditional image synthesis with auxiliary classifier gans. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
17. Ghasedi Dizaji, K., Wang, X., and Huang, H. (2018, January 19–23). Semi-supervised generative adversarial network for gene expression inference. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3220114 ↗
18. Liu, M., Deng, J., Yang, M., Cheng, X., Xie, T., Deng, P., Wang, X., and Liu, M. (2022). Express Construction for GANs from Latent Representation to Data Distribution. Appl. Sci., 12.10.3390/app12083910 ↗
19. Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows10.1145/3447648 ↗
20. Collins, E., Bala, R., Price, B., and Susstrunk, S. (2020, January 13–19). Editing in style: Uncovering the local semantics of gans. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00581 ↗
21. Goetschalckx, L., Andonian, A., Oliva, A., and Isola, P. (November, January 27). Ganalyze: Toward visual definitions of cognitive image properties. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
22. Jahanian, A., Chai, L., and Isola, P. (2019, January 6–9). On the “steerability” of generative adversarial networks. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
23. Shen, Y., Gu, J., Tang, X., and Zhou, B. (2020, January 13–19). Interpreting the latent space of gans for semantic face editing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00926 ↗
24. Generative adversarial nets
25. Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv.
26. Adversarially-learned Inference via an Ensemble of Discrete Undirected Graphical Models
27. Bertran, M., Martinez, N., Papadaki, A., Qiu, Q., Rodrigues, M., Reeves, G., and Sapiro, G. (2019, January 9–15). Adversarially learned representations for information obfuscation and inference. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
28. Berthelot, D., Schumm, T., and Metz, L. (2017). Began: Boundary equilibrium generative adversarial networks. arXiv.
29. Henzler, P., Mitra, N.J., and Ritschel, T. (November, January 27). Escaping Plato’s cave: 3D shape from adversarial rendering. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
30. Gans trained by a two time-scale update rule converge to a local nash equilibrium
31. Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv.
32. Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (2016, January 5–10). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Proceedings of the International Conference on Neural Information Processing Systems, Barcelona, Spain.
33. Radford, A., Metz, L., and Chintala, S. (2015, January 7–9). Unsupervised representation learning with deep convolutional generative adversarial networks. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
34. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. (2017, January 4–9). Improved training of wasserstein GANs. Proceedings of the International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
35. Liao, Y., Schwarz, K., Mescheder, L., and Geiger, A. (2020, January 13–19). Towards unsupervised learning of generative models for 3d controllable image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00591 ↗
36. Hjelm, R.D., Jacob, A.P., Che, T., Trischler, A., Cho, K., and Bengio, Y. (May, January 30). Boundary-seeking generative adversarial networks. Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada.
37. Wu, Z., Lischinski, D., and Shechtman, E. (2021, January 19–25). Stylespace analysis: Disentangled controls for stylegan image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Online.10.1109/CVPR46437.2021.01267 ↗
38. Ganspace: Discovering interpretable gan controls
39. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., and Ng, R. (2020, January 23–28). Nerf: Representing scenes as neural radiance fields for view synthesis. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58452-8_24 ↗
40. Singh, K.K., Ojha, U., and Lee, Y.J. (2019, January 15–20). Finegan: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00665 ↗
41. Voynov, A., and Babenko, A. (2020, January 13–18). Unsupervised discovery of interpretable directions in the gan latent space. Proceedings of the International Conference on Machine Learning, Congress Center, Vienna, Austria.
42. Niemeyer, M., and Geiger, A. (2021, January 19–25). Giraffe: Representing scenes as compositional generative neural feature fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Online.10.1109/CVPR46437.2021.01129 ↗
43. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines10.1080/03610918908812806 ↗
44. Improved techniques for training gans
45. Zhu, X., Xu, C., and Tao, D. (2020). Learning disentangled representations with latent variation predictability. European Conference on Computer Vision, Springer.10.1007/978-3-030-58607-2_40 ↗
46. Karras, T., Aila, T., Laine, S., and Lehtinen, J. (May, January 30). Progressive Growing of GANs for Improved Quality, Stability, and Variation. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.