Enhanced Generative Adversarial Networks with Restart Learning Rate in Discriminator
Source abstract
Abstract
A series of Generative Adversarial Networks (GANs) could effectively capture the salient features in the dataset in an adversarial way, thereby generating target data. The discriminator of GANs provides significant information to update parameters in the generator and itself. However, the discriminator usually becomes converged before the generator has been well trained. Due to this problem, GANs frequently fail to converge and are led to mode collapse. This situation can cause inadequate learning. In this paper, we apply restart learning in the discriminator of the GAN model, which could bring more meaningful updates for the training process. Based on CIFAR-10 and Align Celeba, the experiment results show that the proposed method could improve the performance of a DCGAN with a low FID score over a stable learning rate scheme. Compared with two other stable GANs—SNGAN and WGAN-GP—the DCGAN with a restart schedule had a satisfying performance. Compared with the Two Time-Scale Update Rule, the restart learning rate is more conducive to the training of DCGAN. The empirical analysis indicates four main parameters have varying degrees of influence on the proposed method and present an appropriate parameter setting.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Digital Media Forensic Detection · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Shandong Provincial University Laboratory for Protected Horticulture, Weifang University of Science and Technology, Weifang 262700, China
Department of Computer Engineering, Dongseo University, Busan 47011, Korea
Funding
National Research Foundation of Korea
References (24)
Show source-provided references
1. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K.Q. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems, Curran Associates, Inc.
2. Radford, A., Metz, L., and Chintala, S. (2016, January 2–4). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico.
3. Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6–11). Wasserstein Generative Adversarial Networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
4. Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2017). Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv.
5. Brock, A., Donahue, J., and Simonyan, K. (2019, January 6–9). Large Scale GAN Training for High Fidelity Natural Image Synthesis. Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA.
6. Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. (2019, January 9–15). Self-Attention Generative Adversarial Networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA.
7. Karras, T., Laine, S., and Aila, T. (2019). A Style-Based Generator Architecture for Generative Adversarial Networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE.10.1109/CVPR.2019.00453 ↗
8. Wu, J., Huang, Z., Acharya, D., Li, W., Thoma, J., Paudel, D.P., and Gool, L.V. (2019, January 16–20). Sliced Wasserstein Generative Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00383 ↗
9. Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral Normalization for Generative Adversarial Networks. arXiv.
10. Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (2018). Are GANs Created Equal? A Large-Scale Study. Advances in Neural Information Processing Systems, Curran Associates, Inc.
11. Which Training Methods for GANs do actually Converge?
12. Kurach, K., Lucic, M., Zhai, X., Michalski, M., and Gelly, S. (2019). A Large-Scale Study on Regularization and Normalization in GANs. arXiv.
13. Arjovsky, M., and Bottou, L. (2017, January 24–26). Towards Principled Methods for Training Generative Adversarial Networks. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France. Available online: OpenReview.net.
14. Bińkowski, M., Sutherland, D.J., Arbel, M., and Gretton, A. (2018). Demystifying MMD GANs. arXiv.
15. Roth, K., Lucchi, A., Nowozin, S., and Hofmann, T. (2017). Stabilizing Training of Generative Adversarial Networks through Regularization. arXiv.
16. Wang, B., Nguyen, T.M., Bertozzi, A.L., Baraniuk, R.G., and Osher, S.J. (2020). Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent. arXiv.
17. Smith, L.N. (2017, January 24–31). Cyclical Learning Rates for Training Neural Networks. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.58 ↗
18. Loshchilov, I., and Hutter, F. (2017, January 24–26). SGDR: Stochastic Gradient Descent with Warm Restarts. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France. Available online: OpenReview.net.
19. Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J.E., and Weinberger, K.Q. (2017, January 24–26). Snapshot Ensembles: Train 1, Get M for Free. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France. Available online: OpenReview.net.
20. Li, X., Xiong, H., An, H., Xu, C.Z., and Dou, D. (2020, January 12–18). RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr. Proceedings of the 37th International Conference on Machine Learning, Online.
21. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K.Q. (2014). Identifying and Attacking the Saddle Point Problem in High-dimensional Non-convex Optimization. Advances in Neural Information Processing Systems, Curran Associates, Inc.
22. Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv.
23. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. (2017). Improved Training of Wasserstein GANs. arXiv.
24. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems, Curran Associates, Inc.