FreeMix: Personalized Structure and Appearance Control Without Finetuning
Source abstract
Abstract
Personalized image generation has gained significant attention with the advancement of text-to-image diffusion models. However, existing methods face challenges in effectively mixing multiple visual attributes, such as structure and appearance, from separate reference images. Finetuning-based methods are time-consuming and prone to overfitting, while finetuning-free approaches often suffer from feature entanglement, leading to distortions. To address these challenges, we propose FreeMix, a finetuning-free approach for multi-concept mixing in personalized image generation. Given separate references for structure and appearance, FreeMix generates a new image that integrates both. This is achieved through Disentangle-Mixing Self-Attention (DMSA). DMSA first disentangles the two concepts by applying spatial normalization to remove residual appearance from structure features, and then selectively injects appearance details via self-attention, guided by a cross-attention-derived mask to prevent background leakage. This mechanism ensures precise structural preservation and faithful appearance transfer. Extensive qualitative and quantitative experiments demonstrate that our method achieves superior structural consistency and appearance transfer compared to existing approaches. In addition to personalization, FreeMix can be adapted to exemplar-based image editing.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Computer Graphics and Visualization Techniques · Face recognition and analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence, Hanyang University, Seoul 04763, Republic of Korea
Department of Computer Science and Engineering, Hanyang University, Seoul 04763, Republic of Korea
Funding
Institute of Information and communications Technology Planning and evaluation (IITP) · Korean Government (MSIT)
References (33)
Show source-provided references
1. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022, January 18–24). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01042 ↗
2. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. (2022). Hierarchical text-conditional image generation with clip latents. arXiv.
3. Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., and Cohen-Or, D. (2022). An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv.
4. Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. (2023, January 17–24). Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02155 ↗
5. Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.Y. (2023, January 17–24). Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00192 ↗
6. Voynov, A., Chu, Q., Cohen-Or, D., and Aberman, K. (2023). p+: Extended textual conditioning in text-to-image generation. arXiv.
7. Ding, G., Zhao, C., Wang, W., Yang, Z., Liu, Z., Chen, H., and Shen, C. (2024, January 16–22). FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00868 ↗
8. Hertz, A., Voynov, A., Fruchter, S., and Cohen-Or, D. (2024, January 16–22). Style aligned image generation via shared attention. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00457 ↗
9. Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., and Metaxas, D.N. (2017, January 22–29). Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.629 ↗
10. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. (2018, January 18–23). Attngan: Fine-grained text to image generation with attentional generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00143 ↗
11. Diffusion models beat gans on image synthesis
12. Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. (2021). Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv.
13. Photorealistic text-to-image diffusion models with deep language understanding
14. Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. (2023). Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv.
15. Zhang, L., Rao, A., and Agrawala, M. (2023, January 1–6). Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00355 ↗
16. A neural space-time representation for text-to-image personalization10.1145/3618322 ↗
17. Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. (2023). Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv.
18. Wei, Y., Zhang, Y., Ji, Z., Bai, J., Zhang, L., and Zuo, W. (2023, January 1–6). Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01461 ↗
19. Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. (2022). Prompt-to-prompt image editing with cross attention control. arXiv.
20. Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., and Zheng, Y. (2023, January 1–6). Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.02062 ↗
21. Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. (2023, January 17–24). Plug-and-play diffusion features for text-driven image-to-image translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00191 ↗
22. Brooks, T., Holynski, A., and Efros, A.A. (2023, January 17–24). Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01764 ↗
23. Ju, X., Zeng, A., Bian, Y., Liu, S., and Xu, Q. (2023). Direct inversion: Boosting diffusion-based editing with 3 lines of code. arXiv.
24. Cheng, T.Y., Sharma, P., Markham, A., Trigoni, N., and Jampani, V. (2024). Zest: Zero-shot material transfer from a single image. Computer Vision—ECCV 2024, Proceedings of the 18th European Conference, Milan, Italy, 29 September–4 October 2024, Springer.10.1007/978-3-031-73232-4_21 ↗
25. Denoising diffusion probabilistic models
26. Song, J., Meng, C., and Ermon, S. (2020). Denoising diffusion implicit models. arXiv.
27. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023, January 1–6). Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00371 ↗
28. Huang, X., and Belongie, S. (2017, January 22–29). Arbitrary style transfer in real-time with adaptive instance normalization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.167 ↗
29. Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., and Wen, F. (2023, January 17–24). Paint by example: Exemplar-based image editing with diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01763 ↗
30. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtual.
31. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., and El-Nouby, A. (2023). Dinov2: Learning robust visual features without supervision. arXiv.
32. Tumanyan, N., Bar-Tal, O., Bagon, S., and Dekel, T. (2022, January 18–24). Splicing vit features for semantic appearance transfer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01048 ↗
33. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗