Poe2CLP: phrase-level attention and cross-modal semantic alignment for poem generate chinese landscape paintings
Source abstract
Abstract
Generating traditional Chinese landscape paintings from classical poetry is a challenging cross-modal task due to the condensed semantics and esthetic abstraction of poetic language. Existing text-to-image models struggle to interpret classical Chinese syntax and reproduce ink-wash artistic styles. We propose Poe2CLP, a phrase-level attention and cross-modal alignment framework that dynamically captures composite semantic units in poems and adaptively fuses global mood with local imagery. Built upon a LoRA-enhanced diffusion model and trained on Poegraph-a new dataset of 5200 poem-painting pairs-Poe2CLP outperforms state-of-the-art methods in FID (130.95), CLIP-T (40.75), and CLIP Style Score (0.348). Our approach advances the digital interpretation of East Asian poetic-visual traditions. The dataset and code are publicly available.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Aesthetic Perception and Analysis · Generative Adversarial Networks and Image Synthesis · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (45)
Show source-provided references
1. Xia, C. & Jing, Z. English translation of classical chinese poetry. Orb. Lit. 73, 361–373 (2018).10.1111/oli.12184 ↗
2. Shi, J., Xiong, W., Lin, Z. & Jung, H. J. Instantbooth: Personalized text-to-image generation without test-time finetuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8543–8552 (2024).10.1109/CVPR52733.2024.00816 ↗
3. Dhariwal, P. & Nichol, A. Diffusion models beat GANs on image synthesis. Adv. Neural Inf. Process. Syst. 34, 8780–8794 (2021).
4. Ramesh, A. et al. Zero-shot text-to-image generation. In International Conference on Machine Learning, 8821–8831 (Pmlr, 2021).
5. Wang, X., Liu, X., Wang, H. & Wu, B. A dynamic pre-trained model for Chinese classical poetry. In International Conference on Database Systems for Advanced Applications, 436–445 (Springer, 2024).10.1007/978-981-97-5779-4_30 ↗
6. Wei, Y., Zhu, Y., Bai, T. & Wu, B. A cross-temporal contrastive disentangled model for ancient Chinese understanding. Neural Netw. 179, 106559 (2024).10.1016/j.neunet.2024.106559 ↗
7. Yu, H., Gao, C., Li, X. & Zhang, L. Ancient Chinese Poetry Collation Based on Bert. Procedia Comput. Sci. 242, 1171–1178 (2024).10.1016/j.procs.2024.08.179 ↗
8. Gao, P. et al. Clip-adapter: Better vision-language models with feature adapters. Int. J. Comput. Vis. 132, 581–595 (2024).10.1007/s11263-023-01891-x ↗
9. Zhang, B., Zhang, P., Dong, X., Zang, Y. & Wang, J. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision, 310–325 (Springer, 2024).10.1007/978-3-031-72983-6_18 ↗
10. Guo, S.-C., Liu, S.-K., Wang, J.-Y., Zheng, W.-M. & Jiang, C.-Y. Clip-driven prototype network for few-shot semantic segmentation. Entropy 25, 1353 (2023).10.3390/e25091353 ↗
11. Chowdhury, S. & Soni, B. R-vqa: A robust visual question answering model. Knowl.-Based Syst. 309, 112827 (2025).10.1016/j.knosys.2024.112827 ↗
12. Chowdhury, S. & Soni, B. Handling language prior and compositional reasoning issues in visual question answering system. Neurocomputing 635, 129906 (2025).10.1016/j.neucom.2025.129906 ↗
13. Lymperaiou, M. & Stamou, G. A survey on knowledge-enhanced multimodal learning. Artif. Intell. Rev. 57, 284 (2024).10.1007/s10462-024-10825-z ↗
14. Chowdhury, S. & Soni, B. Beyond words: Esc-net revolutionizes vqa by elevating visual features and defying language priors. Comput. Intell. 40, e70010 (2024).10.1111/coin.70010 ↗
15. Chowdhury, S. & Soni, B. ENVQA: Improving visual question answering model by enriching the visual feature. Eng. Appl. Artif. Intell. 142, 109948 (2025).10.1016/j.engappai.2024.109948 ↗
16. Yang, A. et al. Chinese clip: Contrastive vision-language pretraining in Chinese. https://github.com/OFA-Sys/Chinese-CLIP (2022).
17. Zheng, J., Liang, M., Yu, Y., Li, Y. & Xue, Z. Knowledge graph enhanced multimodal transformer for image-text retrieval. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), 70–82 (IEEE, 2024).10.1109/ICDE60146.2024.00013 ↗
18. Wan, J. et al. Wumkg: a Chinese painting and calligraphy multimodal knowledge graph. Herit. Sci. 12, 159 (2024).10.1186/s40494-024-01268-4 ↗
19. Hu, Y., Zuo, J., Ceng, X., Wan, Z. & Wang, M. 结合全局对应矩阵和相对位置信息的古汉语实体关系联合抽取 (joint extraction of ancient Chinese entity relations by combining global correspondence matrix and relative position information). In Proceedings of the 22nd Chinese National Conference on Computational Linguistics, 241–252 (2023).
20. Zhang, S., Wang, P., Li, Z., Hou, J. & Hu, Q. Confidence-based syntax encoding network for better ancient Chinese understanding. Inf. Process. Manag. 61, 103616 (2024).10.1016/j.ipm.2023.103616 ↗
21. Ho, J., Jain, A. & Abbeel, P. Denoising diffusion probabilistic models. Adv. Neural Inf. Process. Syst. 33, 6840–6851 (2020).
22. Song, J., Meng, C. & Ermon, S. Denoising Diffusion Implicit Models. In International Conference on Learning Representations. (2021).
23. Nichol, A. Q. & Dhariwal, P. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 8162–8171 (PMLR, 2021).
24. Saharia, C. et al. Photorealistic text-to-image diffusion models with deep language understanding. Adv. Neural Inf. Process. Syst. 35, 36479–36494 (2022).
25. Zhao, S. et al. Uni-controlnet: All-in-one control to text-to-image diffusion models. Adv. Neural Inf. Process. Syst. 36, 11127–11150 (2023).
26. Li, M. et al. Controlnet++: Improving conditional controls with efficient consistency feedback: Project page: liming-ai. github. io/controlnet_plus_plus. In European Conference on Computer Vision, 129–147 (Springer, 2024).10.1007/978-3-031-72667-5_8 ↗
27. Mou, C. et al. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 4296–4304 (2024).10.1609/aaai.v38i5.28226 ↗
28. Zhang, Y., Sidibé, D., Morel, O. & Mériaudeau, F. Deep multimodal fusion for semantic image segmentation: A survey. Image Vis. Comput. 105, 104042 (2021).10.1016/j.imavis.2020.104042 ↗
29. Sun, Y., Min, X., Duan, H. & Zhai, G. How is visual attention influenced by text guidance? database and model. IEEE Trans. Image Process. (2024).10.1109/TIP.2024.3461956 ↗
30. Yu, L., Zhang, H. & Xu, C. Text-guided attention is all you need for zero-shot robustness in vision-language models. Adv. Neural Inf. Process. Syst. 37, 96424–96448 (2024).
31. Li, G. Layout control and semantic guidance with attention loss backward for t2i diffusion model. arXiv preprint arXiv:2411.06692 (2024).
32. Zhang, Y., Yang, M., Zhou, Q. & Wang, Z. Attention calibration for disentangled text-to-image personalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4764–4774 (2024).10.1109/CVPR52733.2024.00456 ↗
33. Thengane, V., Khan, S., Hayat, M. & Khan, F. Clip model is an efficient continual learner. arXiv preprint arXiv:2210.03114 (2022).
34. Yan, S., Dong, N., Zhang, L. & Tang, J. Clip-driven fine-grained text-image person re-identification. IEEE Trans. Image Process. 32, 6032–6046 (2023).10.1109/TIP.2023.3327924 ↗
35. Materzyńska, J., Torralba, A. & Bau, D. Disentangling visual and written concepts in clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16410–16419 (2022).10.1109/CVPR52688.2022.01592 ↗
36. Chowdhury, S. & Soni, B. Qsfvqa: A time efficient, scalable and optimized VQA framework. Arab. J. Sci. Eng. 48, 10479–10491 (2023).10.1007/s13369-023-07661-8 ↗
37. Zhang, J. et al. Fengshenbang 1.0: Being the foundation of Chinese cognitive intelligence. CoRR abs/2209.02970 (2022).
38. Che, W., Feng, Y., Qin, L. & Liu, T. N-LTP: An open-source neural language technology platform for Chinese. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 42–49 (Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021). https://aclanthology.org/2021.emnlp-demo.6.10.18653/v1/2021.emnlp-demo.6 ↗
39. Radford, A. et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 8748–8763 (PmLR, 2021).
40. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. & Hochreiter, S. Gans trained by a two-time-scale update rule converge to a local Nash equilibrium. Adv. Neural Inf. Process. Syst. 30 (2017).
41. Zhang, L., Rao, A. & Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3836–3847 (2023).10.1109/ICCV51070.2023.00355 ↗
42. Podell, D. et al. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis[J]. 2024.
43. Wu, X. et al. Taiyi-diffusion-xl: advancing bilingual text-to-image generation with large vision-language model support. arXiv preprint arXiv:2401.14688 (2024).
44. Wang, R. et al. Compositional text-to-image synthesis with attention map control of diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, 5544–5552 (2024).10.1609/aaai.v38i6.28364 ↗
45. Peng, X. et al. Fhs-adapter: fine-grained hierarchical semantic adapter for Chinese landscape paintings generation. Herit. Sci. 12, 265 (2024).10.1186/s40494-024-01370-7 ↗