SKTNet: a semantically guided attribute disentanglement network for fashion sketch editing
Source abstract
Abstract
Fashion sketch editing is intended to modify specific attributes of a sketch while preserving its original integrity, thus facilitating the rapid transformation of designers’ concepts into tangible designs. When fashion sketches are edited, it is crucial to precisely control the style of different parts and ensure that line connections and transitions are smooth and natural. The presence of complex and diverse semantic attributes in fashion sketches poses a challenge to focus existing attribute editing efforts on specific regions. To overcome this limitation, a semantically guided fashion sketch disentanglement model is proposed. First, the complete fashion sketch undergoes semantic segmentation into sleeves and torso using a semantic segmentation network. Thereafter, the latent space of the network is decomposed into semantic parts based on semantic segmentation to prevent editing operations from having unnecessary or unintended effects on non-targeted regions of the fashion sketch. Subsequently, a VGG-structured encoder and StyleGAN2 decoder are trained to obtain the latent vectors of both the complete and segmented sketches. More concise and explanatory features are then extracted in the latent space through sparse principal component analysis. Finally, perturbations along the principal directions are applied to explore variations in attributes related to sleeves and torso in fashion sketches. Extensive qualitative and quantitative experiments on the VITON-HD and Dress-Code datasets demonstrate that our model exhibits outstanding disentanglement ability and produces excellent editing effects in the target attribute regions while keeping the non-target regions virtually unaltered. Furthermore, the attribute disentanglement accuracy is significantly higher than that of other methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · 3D Shape Modeling and Analysis · Handwritten Text Recognition Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Donghua University
Funding
Shanghai Frontier Science Research Center for Modern Textiles, Donghua University · the National Key R&D Program of China
References (49)
Show source-provided references
1. Goodfellow I Pouget-Abadie J Mirza M et al. Generative adversarial nets. In Advances in Neural Information Processing Systems 2014 vol. 27.
2. Zhu JY Krähenbühl P Shechtman E et al. Generative visual manipulation on the natural image manifold. In Computer Vision–ECCV 2016: 14th European Conference Amsterdam The Netherlands October 11–14 2016 Proceedings Part V 14. Dordrecht: Springer International Publishing 2016 pp. 597–613.10.1007/978-3-319-46454-1_36 ↗
3. Li Z Zhou H Bai S et al. M6-Fashion: high-fidelity multi-modal image generation and editing. arXiv preprint arXiv:2205.11705 2022.
4. Chen L Tian J Li G et al. TailorGAN: making user-defined fashion designs. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2020 pp. 3241–3250.10.1109/WACV45572.2020.9093416 ↗
5. Kingma DP Welling M. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 2013.
6. Rezende D Mohamed S. Variational inference with normalizing flows. In International Conference on Machine Learning (PMLR) 2015 pp. 1530–1538.
7. Abdal R Qin Y Wonka P. Image2styleGAN: How to embed images into the styleGAN latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision 2019 pp. 4432–4441.10.1109/ICCV.2019.00453 ↗
8. 10.1109/TNNLS.2018.287519410.1109/TNNLS.2018.2875194 ↗
9. Dai Q Yang S Wang W et al. Edit like a designer: modeling design workflows for unaligned fashion editing. In Proceedings of the 29th ACM International Conference on Multimedia. New York: ACM Press 2021 pp. 3492–3500.10.1145/3474085.3475511 ↗
10. Richardson E Alaluf Y Patashnik O et al. Encoding in style: a styleGAN encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2021 pp. 2287–2296.10.1109/CVPR46437.2021.00232 ↗
11. 10.1109/TIP.2022.316730510.1109/TIP.2022.3167305 ↗
12. Ak KE Lim JH Tham JY et al. Attribute manipulation generative adversarial networks for fashion images. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2019 pp. 10541–10550.10.1109/ICCV.2019.01064 ↗
13. Zhao H Shi J Qi X et al. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2017 pp. 2881–2890.10.1109/CVPR.2017.660 ↗
14. 10.1109/TPAMI.2016.264461510.1109/TPAMI.2016.2644615 ↗
15. Peng C Zhang X Yu G et al. Large kernel matters–improve semantic segmentation by global convolutional network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2017 pp. 4353–4361.10.1109/CVPR.2017.189 ↗
16. Chen LC. Semantic image segmentation with deep convolutional nets and fully connected CRFs. arXiv preprint arXiv:1412.7062 2014.
17. Wei Z Chen L Jin Y et al. Stronger fewer and superior: harnessing vision foundation models for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 pp. 28619–28630.10.1109/CVPR52733.2024.02704 ↗
18. Wu W Mao A Yan W et al. UFS-Net: unsupervised network for fashion style editing and generation. In 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE 2023 pp. 2105–2110.10.1109/ICME55011.2023.00360 ↗
19. 10.1016/j.ipm.2023.10354010.1016/j.ipm.2023.103540 ↗
20. Ping Q Wu B Ding W et al. Fashion-AttGAN: attribute-aware fashion editing with multi-objective GAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops 2019.10.1109/CVPRW.2019.00044 ↗
21. 10.1109/TIP.2019.291675110.1109/TIP.2019.2916751 ↗
22. Kwon Y Petrangeli S Kim D et al. Tailor me: an editing network for fashion attribute shape manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2022 pp. 3831–3840.10.1109/WACV51458.2022.00320 ↗
23. Wang Q Qian J Zou X et al. Coarse-to-fine attribute editing for fashion images. In Artificial Intelligence: First CAAI International Conference CICAI 2021 Hangzhou China June 5–6 2021 Proceedings Part I 1. Dordrecht: Springer International Publishing 2021 pp. 396–407.10.1007/978-3-030-93046-2_34 ↗
24. Dai Q Yang S Wang W et al. Edit like a designer: modeling design workflows for unaligned fashion editing. In Proceedings of the 29th ACM International Conference on Multimedia. New York: ACM Press 2021 pp. 3492–3500.10.1145/3474085.3475511 ↗
25. Pumarola A Agudo A Martinez AM et al. GANimation: anatomically-aware facial animation from a single image. In Proceedings of the European Conference on Computer Vision (ECCV) 2018 pp. 818–833.10.1007/978-3-030-01249-6_50 ↗
26. Zhang G Kan M Shan S et al. Generative adversarial network with spatial attention for face attribute editing. In Proceedings of the European Conference on Computer Vision (ECCV) 2018 pp. 417–432.10.1007/978-3-030-01231-1_26 ↗
27. Kumar A Sattigeri P Balakrishnan A. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848 2017.
28. Higgins I Matthey L Pal A et al. Beta-VAE: Learning basic visual concepts with a constrained variational framework. ICLR (Poster) 2017 3.
29. Chen R TQ Xuechen L Roger G et al. Isolating sources of disentanglement in VAEs. In Proceedings of the 32nd International Conference on Neural Information Processing Systems 2018 pp. 2615–2625.
30. Choi Y Choi M Kim M et al. StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 pp. 8789–8797.10.1109/CVPR.2018.00916 ↗
31. Shi Y Yang X Wan Y et al. SemanticStyleGAN: learning compositional generative priors for controllable image synthesis and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022 pp. 11254–11264.10.1109/CVPR52688.2022.01097 ↗
32. Jahanian A Chai L Isola P. On the “steerability” of generative adversarial networks. arXiv preprint arXiv:1907.07171 2019.
33. Plumerault A Borgne HL Hudelot C. Controlling generative models with continuous factors of variations. arXiv preprint arXiv:2001.10238 2020.
34. Shen Y Gu J Tang X et al. Interpreting the latent space of GANs for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020: pp. 9243–9252.10.1109/CVPR42600.2020.00926 ↗
35. Voynov A Babenko A. Unsupervised discovery of interpretable directions in the GAN latent space. In International Conference on Machine Learning PMLR 2020 pp. 9786–9796.
36. Härkönen E Hertzmann A Lehtinen J et al. GANSpace: discovering interpretable GAN controls. In Advances in Neural Information Processing Systems 2020 vol. 33 pp. 9841–9850.
37. Shen Y Zhou B. Closed-form factorization of latent semantics in GANs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2021 pp. 1532–1540.10.1109/CVPR46437.2021.00158 ↗
38. Zhu J Shen Y Xu Y et al. Region-based semantic factorization in GANs. arXiv preprint arXiv:2202.09649 2022.
39. Wei Y Shi Y Liu X et al. Orthogonal Jacobian regularization for unsupervised disentanglement in image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2021 pp. 6721–6730.10.1109/ICCV48922.2021.00665 ↗
40. 10.1145/3450626.345983810.1145/3450626.3459838 ↗
41. Karras T Laine S Aila T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2019 pp. 4401–4410.10.1109/CVPR.2019.00453 ↗
42. Karras T Laine S Aittala M et al. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020 pp. 8110–8119.10.1109/CVPR42600.2020.00813 ↗
43. Karras T Aittala M Laine S et al. Alias-free generative adversarial networks. In Advances in Neural Information Processing Systems 2021 vol. 34 pp. 852–863.
44. Liu Z Lin Y Cao Y et al. SWIN Transformer: hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2021 pp. 10012–10022.10.1109/ICCV48922.2021.00986 ↗
45. Dosovitskiy A Beyer L Kolesnikov A et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
46. Heusel M Ramsauer H Unterthiner T et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems 2017 vol. 30.
47. Xie S Tu Z. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision 2015 pp. 1395–1403.10.1109/ICCV.2015.164 ↗
48. Martin D Fowlkes C Tal D et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision ICCV 2001. IEEE 2001 vol. 2 pp. 416–423.10.1109/ICCV.2001.937655 ↗
49. He K Zhang X Ren S et al. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2016 pp. 770–778.10.1109/CVPR.2016.90 ↗