CoFaDiff: Coordinating Identity Fidelity and Text Consistency in Diffusion-Based Face Generation
Source abstract
Abstract
Personalized face image generation is essential for Artificial Intelligence-Generated Content (AIGC) applications such as personalized digital avatars and user-customized media creation. However, existing diffusion-based approaches still suffer from insufficient identity consistency and limited text editability. In this work, we propose CoFaDiff, a diffusion-based face generation framework designed for coordinating identity consistency and text-driven editability. To enhance identity consistency, our method integrates a dual-encoder scheme that jointly leverages CLIP and ArcFace to capture both semantic and discriminative facial features, incorporates a progressive curriculum learning strategy to obtain more robust identity representations, and adopts a hybrid IdentityNet–IPAdapter architecture that explicitly models facial location, pose, and corresponding identity embeddings in a unified manner. To enhance text-driven editability, we introduce three complementary optimization strategies: First, long-prompt fine-tuning is employed to reduce the model’s dependency on identity conditions. Second, a semantic alignment loss is incorporated to regularize the influence of identity embeddings within the semantic space of the pretrained diffusion model. Third, during classifier-free guided sampling, we modulate the strength of the identity condition by stacking different numbers of zero-valued identity tokens, enabling users to flexibly balance identity consistency and text editability according to their needs. Experiments on FFHQ and IMDB-WIKI demonstrate that CoFaDiff achieves superior identity consistency and text editability compared to recent baselines.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Face recognition and analysis · Face Recognition and Perception
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
University of Chinese Academy of Sciences, Beijing 101408, China
References (33)
Show source-provided references
1. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022, January 19–24). High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01042 ↗
2. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. (2022). Hierarchical text-conditional image generation with CLIP latents. arXiv.
3. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 25–29). LoRA: Low-rank adaptation of large language models. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Event.
4. Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., and Cohen-Or, D. (2022). An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv.
5. Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. (2023, January 18–22). DreamBooth: Fine-tuning text-to-image diffusion models for subject-driven generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.02155 ↗
6. Papantoniou, F.P., Lattas, A., Moschoglou, S., Deng, J., Kainz, B., and Zafeiriou, S. (2024, January 7–11). Arc2Face: A foundation model for ID-consistent human faces. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.10.1007/978-3-031-72913-3_14 ↗
7. Huang, J., Dong, X., Song, W., Chong, Z., Tang, Z., Zhou, J., Cheng, Y., Chen, L., Li, H., and Yan, Y. (2024). ConsistentID: Portrait generation with multimodal fine-grained identity preserving. arXiv.
8. He, J., Geng, Y., and Bo, L. (2025, January 19–25). UniPortrait: A unified framework for identity-preserving single- and multi-human image personalization. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Honolulu, HI, USA.
9. Li, Z., Cao, M., Wang, X., Qi, Z., Cheng, M.-M., and Shan, Y. (2024, January 17–21). PhotoMaker: Customizing realistic human photos via stacked ID embedding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00825 ↗
10. Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. (2023). IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv.
11. PuLID: Pure and lightning ID customization via contrastive alignment
12. Zhou, Z., Li, J., Li, H., Chen, N., and Tang, X. (2024). StoryMaker: Towards holistic consistent characters in text-to-image generation. arXiv.
13. Wang, Q., Bai, X., Wang, H., Qin, Z., Chen, A., Li, H., Tang, X., and Hu, Y. (2024). InstantID: Zero-shot identity-preserving generation in seconds. arXiv.
14. Shiohara, K., and Yamasaki, T. (2024, January 17–21). Face2Diffusion for fast and editable face personalization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00654 ↗
15. Valevski, D., Lumen, D., Matias, Y., and Leviathan, Y. (2023, January 12–15). Face0: Instantaneously conditioning a text-to-image model on a face. Proceedings of the SIGGRAPH Asia 2023 Conference Papers, Sydney, Australia.10.1145/3610548.3618249 ↗
16. FastComposer: Tuning-free multi-subject image generation with localized attention10.1007/s11263-024-02227-z ↗
17. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning (ICML), Virtual Event.
18. Deng, J., Guo, J., Xue, N., and Zafeiriou, S. (2019, January 16–20). ArcFace: Additive angular margin loss for deep face recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00482 ↗
19. Zhang, L., Rao, A., and Agrawala, M. (2023, January 2–6). Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00355 ↗
20. Denoising diffusion probabilistic models
21. Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., and Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv.
22. Nichol, A.Q., and Dhariwal, P. (2021, January 18–24). Improved denoising diffusion probabilistic models. Proceedings of the International Conference on Machine Learning (ICML), Virtual Event.
23. Diffusion models beat GANs on image synthesis
24. Photorealistic text-to-image diffusion models with deep language understanding
25. Huang, Y., Yi, H., Liu, W., Wang, H., Wu, B., Wang, W., Lin, B., Zhang, D., and Cai, D. (2023, January 2–6). One-shot implicit animatable avatars with model-based priors. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00824 ↗
26. Yan, Y., Zhang, C., Wang, R., Zhou, Y., Zhang, G., Cheng, P., Yu, G., and Fu, B. (2023). FaceStudio: Put your face everywhere in seconds. arXiv.
27. Li, J., Li, D., Savarese, S., and Hoi, S. (2023, January 23–29). BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. Proceedings of the International Conference on Machine Learning (ICML), Honolulu, HI, USA.
28. Visual instruction tuning
29. Karras, T., Laine, S., and Aila, T. (2019, January 16–20). A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00453 ↗
30. Rothe, R., Timofte, R., and Van Gool, L. (2015, January 7–13). DEX: Deep expectation of apparent age from a single image. Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), Santiago, Chile.10.1109/ICCVW.2015.41 ↗
31. Parmar, G., Zhang, R., and Zhu, J.-Y. (2022, January 19–24). On aliased resizing and surprising subtleties in GAN evaluation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01112 ↗
32. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Minneapolis, MN, USA.
33. Lin, S., Wang, A., and Yang, X. (2024). SDXL-Lightning: Progressive adversarial diffusion distillation. arXiv.