Masked Style Transfer for Source-Coherent Image-to-Image Translation
Source abstract
Abstract
The goal of image-to-image translation (I2I) is to translate images from one domain to another while maintaining the content representations. A popular method for I2I translation involves the use of a reference image to guide the transformation process. However, most architectures fail to maintain the input’s main characteristics and produce images that are too similar to the reference during style transfer. In order to avoid this problem, we propose a novel architecture that is able to perform source-coherent translation between multiple domains. Our goal is to preserve the input details during I2I translation by weighting the style code obtained from the reference images before applying it to the source image. Therefore, we choose to mask the reference images in an unsupervised way before extracting the style from them. By doing so, the input characteristics are better maintained while performing the style transfer. As a result, we also increase the diversity in the generated images by extracting the style from the same reference. Additionally, adaptive normalization layers, which are commonly used to inject styles into a model, are substituted with an attention mechanism for the purpose of increasing the quality of the generated images. Several experiments are performed on the CelebA-HQ and AFHQ datasets in order to prove the efficacy of the proposed system. Quantitative results measured using the LPIPS and FID metrics demonstrate the superiority of the proposed architecture compared to the state-of-the-art methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Engineering and Architecture, University of Parma, 43124 Parma, Italy
References (37)
Show source-provided references
1. Image-to-Image Translation: Methods and Applications10.1109/TMM.2021.3109419 ↗
2. Li, Y.A., Zare, A., and Mesgarani, N. (2021). StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion. arXiv.10.21437/Interspeech.2021-319 ↗
3. Zhu, J., Park, T., Isola, P., and Efros, A.A. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. arXiv.10.1109/ICCV.2017.244 ↗
4. Zheng, Z., Wu, Y., Han, X., and Shi, J. (2020, January 23–28). ForkGAN: Seeing into the Rainy Night. Proceedings of the IEEE European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58580-8_10 ↗
5. Choi, Y., Uh, Y., Yoo, J., and Ha, J. (2019). StarGAN v2: Diverse Image Synthesis for Multiple Domains. arXiv.10.1109/CVPR42600.2020.00821 ↗
6. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2021). High-Resolution Image Synthesis with Latent Diffusion Models. arXiv.10.1109/CVPR52688.2022.01042 ↗
7. Huang, X., and Belongie, S.J. (2017). Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. arXiv.10.1109/ICCV.2017.167 ↗
8. Hamilton, M., Zhang, Z., Hariharan, B., Snavely, N., and Freeman, W.T. (2022). Unsupervised Semantic Segmentation by Distilling Feature Correspondences. arXiv.
9. Isola, P., Zhu, J., Zhou, T., and Efros, A.A. (2016). Image-to-Image Translation with Conditional Adversarial Networks. arXiv.10.1109/CVPR.2017.632 ↗
10. Mirza, M., and Osindero, S. (2014). Conditional Generative Adversarial Nets. arXiv.
11. Huang, X., Liu, M., Belongie, S.J., and Kautz, J. (2018). Multimodal Unsupervised Image-to-Image Translation. arXiv.10.1007/978-3-030-01219-9_11 ↗
12. Mao, Q., Lee, H., Tseng, H., Ma, S., and Yang, M. (2019). Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis. arXiv.10.1109/CVPR.2019.00152 ↗
13. Choi, Y., Choi, M., Kim, M., Ha, J., Kim, S., and Choo, J. (2017). StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. arXiv.10.1109/CVPR.2018.00916 ↗
14. Ho, J., Jain, A., and Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. arXiv.
15. Zhang, L., Rao, A., and Agrawala, M. (2023). Adding Conditional Control to Text-to-Image Diffusion Models. arXiv.10.1109/ICCV51070.2023.00355 ↗
16. Li, B., Xue, K., Liu, B., and Lai, Y.K. (2023, January 17–24). Bbdm: Image-to-image translation with brownian bridge diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00194 ↗
17. Saharia, C., Chan, W., Chang, H., Lee, C.A., Ho, J., Salimans, T., Fleet, D.J., and Norouzi, M. (2021). Palette: Image-to-Image Diffusion Models. arXiv.10.1145/3528233.3530757 ↗
18. Gatys, L.A., Ecker, A.S., and Bethge, M. (July, January 26). Image Style Transfer Using Convolutional Neural Networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.265 ↗
19. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
20. Deng, Y., Tang, F., Pan, X., Dong, W., Ma, C., and Xu, C. (2021). StyTr^2: Unbiased Image Style Transfer with Transformers. arXiv.10.1109/CVPR52688.2022.01104 ↗
21. Chung, J., Hyun, S., and Heo, J.P. (2024, January 17–21). Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00840 ↗
22. Deng, Y., He, X., Tang, F., and Dong, W. (2024, January 17–21). Z*: Zero-shot Style Transfer via Attention Reweighting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00662 ↗
23. Mejjati, Y.A., Richardt, C., Tompkin, J., Cosker, D., and Kim, K.I. (2018). Unsupervised Attention-guided Image to Image Translation. arXiv.
24. Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., and Tombari, F. (2022, January 23–27). Avoiding Shortcuts in Unpaired Image-to-Image Translation. Proceedings of the Image Analysis and Processing—ICIAP 2022, Lecce, Italy.10.1007/978-3-031-06430-2 ↗
25. Zhu, P., Abdal, R., Qin, Y., and Wonka, P. (2019). SEAN: Image Synthesis with Semantic Region-Adaptive Normalization. arXiv.10.1109/CVPR42600.2020.00515 ↗
26. He, K., Gkioxari, G., Dollár, P., and Girshick, R.B. (2017). Mask R-CNN. arXiv.10.1109/ICCV.2017.322 ↗
27. Redmon, J., Divvala, S.K., Girshick, R.B., and Farhadi, A. (2015). You Only Look Once: Unified, Real-Time Object Detection. arXiv.10.1109/CVPR.2016.91 ↗
28. Ji, X., Henriques, J.F., and Vedaldi, A. (2018). Invariant Information Distillation for Unsupervised Image Segmentation and Clustering. arXiv.10.1109/ICCV.2019.00996 ↗
29. Cho, J.H., Mall, U., Bala, K., and Hariharan, B. (2021). PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering. arXiv.
30. Fontanini, T., and Ferrari, C. (2022). Would Your Clothes Look Good on Me? Towards Transferring Clothing Styles with Adaptive Instance Normalization. Sensors, 22.10.3390/s22135002 ↗
31. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerging Properties in Self-Supervised Vision Transformers. arXiv.10.1109/ICCV48922.2021.00951 ↗
32. Krähenbühl, P., and Koltun, V. (2012). Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. arXiv.
33. Liu, Z., Luo, P., Wang, X., and Tang, X. (2015, January 7–13). Deep Learning Face Attributes in the Wild. Proceedings of the International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.425 ↗
34. Caesar, H., Uijlings, J.R.R., and Ferrari, V. (2016). COCO-Stuff: Thing and Stuff Classes in Context. arXiv.
35. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
36. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., and Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium. arXiv.
37. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. arXiv.10.1109/CVPR.2018.00068 ↗