Bootstrapped SSL CycleGAN for Asymmetric Domain Transfer
Source abstract
Abstract
Most CycleGAN domain transfer architectures require a large amount of data belonging to domains on which the domain transfer task is to be applied. Nevertheless, in many real-world applications one of the domains is reduced, i.e., scarce. This means that it has much less training data available in comparison to the other domain, which is fully observable. In order to tackle the problem of using CycleGAN framework in such unfavorable application scenarios, we propose and invoke a novel Bootstrapped SSL CycleGAN architecture (BTS-SSL), where the mentioned problem is overcome using two strategies. Firstly, by using a relatively small percentage of available labelled training data from the reduced or scarce domain and a Semi-Supervised Learning (SSL) approach, we prevent overfitting of the discriminator belonging to the reduced domain, which would otherwise occur during initial training iterations due to the small amount of available training data in the scarce domain. Secondly, after initial learning guided by the described SSL strategy, additional bootstrapping (BTS) of the reduced data domain is performed by inserting artifically generated training examples into the training poll of the data discriminator belonging to the scarce domain. Bootstrapped samples are generated by the already trained neural network that performs transferring from the fully observable to the scarce domain. The described procedure is periodically repeated during the training process several times and results in significantly improved performance of the final model in comparison to the original unsupervised CycleGAN approach. The same also holds in comparison to the solutions that are exclusively based either on the described SSL, or on the bootstrapping strategy, i.e., when these are applied separately. Moreover, in the considered scarce scenarios it also shows competitive results in comparison to the fully supervised solution based on the pix2pix method. In that sense, it is directly applicable to many domain transfer tasks that are relying on the CycleGAN architecture.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Processing Techniques · Image Enhancement Techniques · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Technical Sciences, University of Novi Sad, Trg D. Obradovića 6, 21000 Novi Sad, Serbia
Institute of Mathematics, Serbian Academy of Sciences and Arts, Kneza Mihaila 36, 11000 Belgrade, Serbia
Funding
Science Fund of the Republic of Serbia · Serbian Ministry of Education, Science and Technological Development
References (28)
Show source-provided references
1. Park, T., Liu, M.Y., Wang, T.C., and Zhu, J.Y. (2019, January 15–20). Semantic image synthesis with spatially-adaptive normalization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00244 ↗
2. Zhu, P., Abdal, R., Qin, Y., and Wonka, P. (2020, January 13–19). SEAN: Image synthesis with semantic region-adaptive normalization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00515 ↗
3. Lee, C.H., Liu, Z., Wu, L., and Luo, P. (2020, January 13–19). MaskGAN: Towards diverse and interactive facial image manipulation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00559 ↗
4. Tang, H., Xu, D., Yan, Y., Torr, P.H., and Sebe, N. (2020, January 13–19). Local class-specific and global image-level generative adversarial networks for semantic-guided scene generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00789 ↗
5. Yi, Z., Zhang, H., Tan, P., and Gong, M. (2017, January 22–29). DualGAN: Unsupervised dual learning for image-to-image translation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.310 ↗
6. Mejjati, Y.A., Richardt, C., Tompkin, J., Cosker, D., and Kim, K.I. (2018, January 3–8). Unsupervised attention-guided image-to-image translation. Proceedings of the 31st International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
7. Tomei, M., Cornia, M., Baraldi, L., and Cucchiara, R. (2019, January 15–20). Art2Real: Unfolding the reality of artworks via semantically-aware image-to-image translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00600 ↗
8. Liu, M.Y., Huang, X., Mallya, A., Karras, T., Aila, T., Lehtinen, J., and Kautz, J. (November, January 27). Few-shot unsupervised image-to-image translation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
9. Song, Y., Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., and Kuo, C.C.J. (2018, January 8–14). Contextual-based image inpainting: Infer, match, and translate. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01216-8_1 ↗
10. Zhao, L., Mo, Q., Lin, S., Wang, Z., Zuo, Z., Chen, H., Xing, W., and Lu, D. (2020, January 13–19). UCTGAN: Diverse image inpainting based on unsupervised cross-space translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00578 ↗
11. Yuan, Y., Liu, S., Zhang, J., Zhang, Y., Dong, C., and Lin, L. (2018, January 18–23). Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00113 ↗
12. Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution10.1109/TIP.2019.2938347 ↗
13. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21–26). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.632 ↗
14. Wang, C., Zheng, H., Yu, Z., Zheng, Z., Gu, Z., and Zheng, B. (2018, January 8–14). Discriminative region proposal adversarial networks for high-quality image-to-image translation. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01246-5_47 ↗
15. Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., and Catanzaro, B. (2018, January 18–23). High-resolution image synthesis and semantic manipulation with conditional GANs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00917 ↗
16. AlBahar, B., and Huang, J.B. (November, January 27). Guided image-to-image translation with bi-directional feature transformation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
17. Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
18. Bao, F., Neumann, M., and Vu, N.T. (2019, January 15–19). CycleGAN-based emotion style transfer as data augmentation for speech emotion recognition. Proceedings of the Conference of the International Speech Communication Association (Interspeech), Graz, Austria.10.21437/Interspeech.2019-2293 ↗
19. Meng, Z., Li, J., Gong, Y., and Juang, B.H. (2018, January 2–6). Cycle-consistent speech enhancement. Proceedings of the Conference of the International Speech Communication Association (Interspeech), Hyderabad, India.10.21437/Interspeech.2018-2409 ↗
20. Hosseini-Asl, E., Zhou, Y., Xiong, C., and Socher, R. (2018, January 3–8). Robust domain adaptation by augmented cyclic adversarial learning. Proceedings of the 31st International Conference on Neural Information Processing Systems—Interpretability and Robustness for Audio, Speech and Language Workshop, Montreal, QC, Canada.
21. Qi, C., Chen, J., Xu, G., Xu, Z., Lukasiewicz, T., and Liu, Y. (2020). SAG-GAN: Semi-supervised attention-guided GANs for data augmentation on medical images. arXiv.
22. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative adversarial nets. Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
23. Ulyanov, D., Vedaldi, A., and Lempitsky, V. (2017). Instance normalization: The missing ingredient for fast stylization. arXiv.
24. Johnson, J., Alahi, A., and Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. European Conference on Computer Vision, Springer.10.1007/978-3-319-46475-6_43 ↗
25. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
26. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. (2016, January 20–22). Generative adversarial text to image synthesis. Proceedings of the International Conference on Machine Learning, PMLR, New York, NY, USA.
27. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
28. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗