An intelligent generative method of fashion design combining attribute knowledge and Stable Diffusion Model
Source abstract
Abstract
Artificial intelligence generation technology has brought new opportunities to the field of fashion design. Attribute knowledge has a significant impact on the overall effect of fashion design. Contemporary generative methods of fashion design frequently yield results lacking semantic information or missing specific attributes. To address the problem, this study aims for an intelligent generative method of fashion design through constructing prompt templates and a specific attribute low-rank adaption (LoRA) to combine fashion attribute knowledge into the generative process of the Stable Diffusion Model. First, a fashion attribute knowledge graph is constructed to establish prompt templates, and natural language descriptions are transformed into professional and complete prompt through GPT-4. Second, the fashion dataset is annotated with templates to filter specific attributes for LoRA training, followed by controlling fashion attributes in generation. Furthermore, analyses of the generation of women’s jacket designs show that the proposed method consistently improves the accuracy and stability of attributes in fashion design generation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Color perception and design · Digital Media and Visual Art · Aesthetic Perception and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Art Design and Media, East China University of Science and Technology, China
Funding
China National Funds for Distinguished Young Scientists · Scientific and Innovative Action Plan of Shanghai
References (65)
Show source-provided references
1. Arar M Voynov A Hertz A et al. PALP: Prompt aligned personalization of text-to-image models. 2024. DOI: 10.48550/arXiv.2401.06105.10.1145/3680528.3687604 ↗
2. Wang W. Research on intelligent design of clothing appearance based on deep learning. Master’s Thesis Zhejiang University China 2022.
3. 10.1007/s10660-023-09776-410.1007/s10660-023-09776-4 ↗
4. Liu L, Zhang H, Zhou D, et al. Toward fashion intelligence in the big data era: state-of-the-art and future prospects. IEEE Trans Consum Electron 2023; 70. doi:10.1109/TCE.2023.3285880.
5. Ping Q Wu B Ding W et al. Fashion-AttGAN: Attribute-aware fashion editing with multi-objective gan. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE 2019 pp. 323–325.10.1109/CVPRW.2019.00044 ↗
6. Kim J Lee JK Lee KM. Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE 2016 pp. 1637–1645.10.1109/CVPR.2016.181 ↗
7. Rombach R Blattmann A Lorenz D et al. High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022 pp. 10684–10695.10.1109/CVPR52688.2022.01042 ↗
8. Choi Y Kwak S Lee K et al. Improving diffusion models for virtual try-on. DOI: 10.48550/arXiv.2403.05139.
9. Xu Y, Gu T, Chen W, et al. Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on. Computer Vision and Pattern Recognition (cs.CV) 2024. doi:10.48550/arXiv.2403.01779.
10. Baldrati A Morelli D Cartella G et al. Multimodal garment designer: Human-centric latent diffusion models for fashion image editing. In: Computer Vision and Pattern Recognition. IEEE 2023 pp. 23393–23402.10.1109/ICCV51070.2023.02138 ↗
11. Zhang X Yang B Kampffmeyer MC et al. Diffcloth: Diffusion based garment synthesis and manipulation via structural cross-modal semantic alignment. IEEE International Conference on Computer Vision 2023 pp. 23154–23163.10.1109/ICCV51070.2023.02116 ↗
12. Huang L Chen D Liu Y et al. Composer: Creative and controllable image synthesis with composable conditions. 2023. doi:10.48550/arXiv.2302.09778.
13. Yang L, Yu Z, Meng C, et al. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. CVPR’19, 2024. doi:10.48550/arXiv.1904.01538
14. Wang M Liu Y Zhang X et al. LangGPT: Rethinking structured reusable prompt design framework for LLMs from the programming language. 2024. doi:arXiv:2402.16929 2024.
15. Wang W-Z Xiao H-M Fang Y. Clothing image attribute editing based on generative adversarial network with reference to an upper garment. Emerald Publishing Limited 2024. doi:10.1108/ijcst-09-2023-0129.10.1108/IJCST-09-2023-0129 ↗
16. 10.1016/j.cad.2004.08.00710.1016/j.cad.2004.08.007 ↗
17. 10.1016/j.cad.2004.09.00310.1016/j.cad.2004.09.003 ↗
18. 10.1016/j.knosys.2017.07.00710.1016/j.knosys.2017.07.007 ↗
19. 10.1016/j.compind.2010.03.00210.1016/j.compind.2010.03.002 ↗
20. 10.1016/j.eswa.2009.05.08910.1016/j.eswa.2009.05.089 ↗
21. 10.1155/2022/333404710.1155/2022/3334047 ↗
22. Yang B. Clothing design style recommendation using decision tree algorithm combined with deep learning. Computat Intell Neurosci 2022. doi:10.1155/2022/5745457.
23. 10.3390/electronics1220426310.3390/electronics12204263 ↗
24. 10.1016/j.knosys.2023.11079110.1016/j.knosys.2023.110791 ↗
25. 10.1007/s10462-021-10039-710.1007/s10462-021-10039-7 ↗
26. 10.1177/155892502092003510.1177/1558925020920035 ↗
27. 10.1145/342262210.1145/3422622 ↗
28. 10.1080/09540091.2020.182278010.1080/09540091.2020.1822780 ↗
29. 10.1016/j.knosys.2021.10687310.1016/j.knosys.2021.106873 ↗
30. 10.34133/icomputing.001710.34133/icomputing.0017 ↗
31. Song J Meng C Ermon S. Denoising diffusion implicit models. 2020. doi:10.48550/arXiv.2010.02502.
32. Dhariwal P, Nichol A. Diffusion models beat GANs on image synthesis. Adv Neural Inform Process Syst 2021; 34: 8780–8794.
33. 10.1145/355472910.1145/3554729 ↗
34. 10.1145/355472910.1145/3554729 ↗
35. Ruiz N Li Y Jampani V et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In: Computer Vision and Pattern Recognition (cs.CV). IEEE 2023 pp. 22500–22510.10.1109/CVPR52729.2023.02155 ↗
36. Hu EJ Shen Y Wallis P et al. Lora: Low-rank adaptation of large language models. 2021. doi:10.48550/arXiv.2106.09685.
37. Cao T Wang C Liu B et al. Beautifulprompt: Towards automatic prompt engineering for text-to-image synthesis. 2023. doi:10.48550/arXiv.2311.06752.10.18653/v1/2023.emnlp-industry.1 ↗
38. Rosenman S Lal V Howard P. NeuroPrompts: An adaptive framework to optimize prompts for text-to-image generation. 2023. doi:10.48550/arXiv.2311.12229.
39. Lee K Liu H Ryu M et al. Aligning text-to-image models using human feedback. 2023. doi:10.48550/arXiv.2302.12192.
40. Daiki Miyake Akihiro Iohara Yu Saito andToshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with textguided diffusion models. arXiv preprint arXiv:2305.16807 2023. doi:10.48550/arXiv.2302.12192.
41. Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. Adv Neural Inform Process Syst 2020; 33: 1877–1901.
42. 10.1016/j.knosys.2023.11092810.1016/j.knosys.2023.110928 ↗
43. Liu Z Luo P Qiu S et al. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In: Computer Vision & Pattern Recognition. IEEE 2016 pp. 1096–1104.10.1109/CVPR.2016.124 ↗
44. Zou X Kong X Wong W et al. Fashionai: A hierarchical dataset for fashion understanding. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE 2019.10.1109/CVPRW.2019.00039 ↗
45. 10.1186/s40691-023-00338-810.1186/s40691-023-00338-8 ↗
46. Bonatti PA Decker S Polleres A et al. Knowledge graphs: new directions for knowledge representation on the semantic web (Dagstuhl seminar 18371). In: Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik. 2019 pp. 29–111.
47. Zhao Y Feng X Zhu M et al. Personalized clothing recommendation method based on knowledge graph. In: IEEE. 2023 pp. 230–237.10.1109/DSIT60026.2023.00042 ↗
48. Zhang T Zhang Y Vineet V et al. Controllable text-to-image generation with GPT-4. 2023. doi: 10.48550/arXiv.2305.18583.
49. Yang Z Li L Lin K et al. The dawn of lMMs: preliminary explorations with GPT-4v (vision) 2023. DOI: 10.48550/arXiv.2309.17421.
50. Kingma DP Welling M. Auto-encoding variational Bayes. 2013. doi:10.48550/1312.6114.
51. Mishchenko K Defazio A. Prodigy: An expeditiously adaptive parameter-free learner. 2023. doi:10.48550/arXiv.2306.06101.
52. AUTOMATIC1111. stable-diffusion-webui. https://github.com/AUTOMATIC1111/stable-diffusion-webui (2023 accessed 31 October 2023).
53. Zhang L Rao A Agrawala M. Adding conditional control to text-to-image diffusion models. In: Computer Vision and Pattern Recognition (cs.CV). IEEE 2023 pp. 3836–3847.10.1109/ICCV51070.2023.00355 ↗
54. Wang T, Yang X, Xu K, et al. Spatial attentive single-image deraining with a high quality real rain dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. doi:10.48550/arXiv.1904.01538
55. Xu Q Huang G Yuan Y et al. An empirical study on evaluation metrics of generative adversarial networks. 2018. doi:10.48550/arXiv.1806.0775510.1109/BigData.2018.8622525 ↗
56. Zhang R Isola P Efros AA et al. The unreasonable effectiveness of deep features as a perceptual metric. 2018. doi:10.1109/CVPR.2018.00068.10.1109/CVPR.2018.00068 ↗
57. 10.1109/LSP.2012.222772610.1109/LSP.2012.2227726 ↗
58. 10.1109/TIP.2012.221405010.1109/TIP.2012.2214050 ↗
59. Cao S, Chai W, Hao S, et al. Difffashion: Reference-based fashion design with structure-aware transfer by diffusion models. IEEE, 2023. doi:10.1109/tmm.2023.3318297.
60. Wang S Chen L Jiang J et al. LoRA meets dropout under a unified framework. 2024. doi:10.48550/arXiv.2403.0081210.18653/v1/2024.findings-acl.119 ↗
61. Datta S Ku A Ramachandran D et al. Prompt expansion for adaptive text-to-image generation. 2023. doi:10.48550/arXiv.2312.16720.
62. Yang L, Zhang Z, Song Y, et al. Diffusion models: A comprehensive survey of methods and applications. ACM Comput Surv 2023. doi:10.1145/3626235
63. Islam K, Zaheer MZ, Mahmood A, et al. DiffuseMix: Label-preserving data augmentation with diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
64. Gong Q, Li X. Research progress on knowledge graph in the field of clothing J Silk; 2023.
65. Argyrou G Dimitriou A Lymperaiou M et al. Automatic Generation of Fashion Images using Prompting in Generative Machine Learning Models. 2024. doi:10.48550/arXiv.2407.14944.