Challenges in Generating Accurate Text in Images: A Benchmark for Text-to-Image Models on Specialized Content
Source abstract
Abstract
Rapid advances in text-to-image (T2I) generative models have significantly enhanced visual content creation. However, evaluating these models remains challenging, particularly when assessing their ability to handle complex textual content. The primary aim of this research is to develop a systematic evaluation framework for assessing T2I models’ capabilities in generating specialized content, with emphasis on measuring text rendering accuracy and identifying model limitations across diverse domains. The framework utilizes carefully crafted prompts that require precise formatting, semantic alignment, and compositional reasoning to evaluate model performance. Our evaluation methodology encompasses a comprehensive assessment across many critical domains: mathematical equations, chemical diagrams, programming code, flowcharts, multi-line text, and paragraphs, with each domain tested through specifically designed challenge sets. GPT-4 serves as an automated evaluator, assessing outputs based on key metrics such as text accuracy, readability, formatting consistency, visual design, contextual relevance, and error recovery. Weighted scores generated by GPT-4 are compared with human evaluations to measure alignment and reliability. The results reveal that current T2I models face significant challenges with tasks requiring structural precision and domain-specific accuracy. Notable difficulties include symbol alignment in equations, bond angles in chemical diagrams, syntactical correctness in code, and the generation of coherent multi-line text and paragraphs. This study advances our understanding of fundamental limitations in T2I model architectures while establishing a novel framework for the systematic evaluation of text rendering capabilities. Despite these limitations, the proposed benchmark provides a clear pathway for evaluating and tracking improvements in T2I models, establishing a standardized framework for assessing their ability to generate accurate and reliable structured content for specialized applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Image Retrieval and Classification Techniques · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Software Engineering, Institute of Informatics, University of Szeged, 6720 Szeged, Hungary
Funding
European Union · Okos Gazda Szeme (IntelliFarmMoo)
References (42)
Show source-provided references
1. Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., and Choi, Y. (2021). CLIPScore: A Reference-Free Evaluation Metric for Image Captioning. arXiv.10.18653/v1/2021.emnlp-main.595 ↗
2. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017, January 4–9). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA.
3. Sampaio, G.G., Zhang, R., Zhai, S., Gu, J., Susskind, J., Jaitly, N., and Zhang, Y. (2024). TypeScore: A Text Fidelity Metric for Text-to-Image Generative Models. arXiv.
4. Lu, Y., Yang, X., Li, X., Wang, X., and Wang, W.Y. (2023). LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation. arXiv.
5. Zhang, X., Lu, Y., Wang, W., Yan, A., Yan, J., Qin, L., Wang, H., Yan, X., Wang, W.Y., and Petzold, L.R. (2023). GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks. arXiv.
6. Tan, Z., Yang, X., Qin, L., Yang, M., Zhang, C., and Li, H. (2024). EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Models. arXiv.
7. Chen, Y., Liu, L., and Ding, C. (2023). X-IQE: EXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Models. arXiv.
8. Ku, M.W., Jiang, D., Wei, C., Yue, X., and Chen, W. (2023). VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation. arXiv.10.18653/v1/2024.acl-long.663 ↗
9. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv.10.18653/v1/2023.emnlp-main.153 ↗
10. Chen, D., Chen, R., Zhang, S., Liu, Y., Wang, Y., Zhou, H., Zhang, Q., Zhou, P., Wan, Y., and Sun, L. (2024). MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark. arXiv.
11. Huang, Z., He, W., Long, Q., Wang, Y., Li, H., Yu, Z., Shu, F., Chan, L., Jiang, H., and Gan, L. (2024). T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts. arXiv.
12. Mi, X., Tang, F., Cao, J., Sheng, Q., Huang, Z., Li, P., Liu, Y., and Lee, T.Y. (2024). Interactive Visual Assessment for Text-to-Image Generation Models. arXiv.
13. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2023). MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv.
14. Tu, R.C., Ma, Z.A., Lan, T., Zhao, Y., Huang, H., and Mao, X.L. (2024). Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark. arXiv.
15. Semantic Object Accuracy for Generative Text-to-Image Synthesis10.1109/TPAMI.2020.3021209 ↗
16. Cho, J., Zala, A., and Bansal, M. (2022). DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers. arXiv.10.1109/ICCV51070.2023.00283 ↗
17. Cho, J., Zala, A., and Bansal, M. (2023, January 2–6). DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00283 ↗
18. Luo, Y., Cheng, M., Ouyang, J., Tao, X., and Liu, Q. (2024). TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization. arXiv.
19. Bakr, E.M., Sun, P., Shen, X., Khan, F.F., Li, L.E., and Elhoseiny, M. (2023, January 2–6). HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01834 ↗
20. Zhao, B., Zhang, L., Wu, J., and Luo, W. (2019, January 15–20). Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.
21. Cheng, J., Liang, X., Shi, X., He, T., Xiao, T., and Li, M. (2023). LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation. arXiv.
22. Huang, J., Huang, M., Zhang, W., Li, X., and Chen, R. (2024). SmartSpatial: Enhancing the 3D Spatial Arrangement in Generative Models. arXiv.
23. Chatterjee, A., Ben Melech Stan, G., Aflalo, E., Paul, S., Ghosh, D., Gokhale, T., Schmidt, L., Hajishirzi, H., Lal, V., and Baral, C. (2024). Getting it Right: Improving Spatial Consistency in Text-to-Image Models. arXiv.10.1007/978-3-031-72670-5_12 ↗
24. Zhang, G., Fu, B., Fan, Q., Zhang, Q., Liu, R., Gu, H., Zhang, H., and Liu, X. (2024). CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Models. arXiv.
25. Lee, T., Yasunaga, M., Meng, C., Mai, Y., Park, J.S., Gupta, A., Zhang, Y., Narayanan, D., Teufel, H., and Bellagente, M. (2023). Holistic Evaluation of Text-To-Image Models. arXiv.
26. Huang, K., Sun, K., Xie, E., Li, Z., and Liu, X. (2023). T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation. arXiv.
27. Ghosh, D., Hajishirzi, H., and Schmidt, L. (2023). GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. arXiv.
28. Li, B., Lin, Z., Pathak, D., Li, J., Fei, Y., Wu, K., Ling, T., Xia, X., Zhang, P., and Neubig, G. (2024). GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation. arXiv.10.1109/CVPRW63382.2024.00538 ↗
29. Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., and Smith, N.A. (2023, January 2–6). TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01866 ↗
30. Lim, Y., Choi, H., Chen, P.Y., and Shim, H. (2024). Evaluating Image Hallucination in Text-to-Image Generation with Question-Answering. arXiv.
31. Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. (2024). Evaluating Text-to-Visual Generation with Image-to-Text Generation. arXiv.10.1007/978-3-031-72673-6_20 ↗
32. Zhu, X., Sun, P., Wang, C., Liu, J., Li, Z., Xiao, Y., and Huang, J. (2023). A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metrics. arXiv.
33. Foka, A. (2024). A Framework for Critical Evaluation of Text-to-Image Models: Integrating Art Historical Analysis, Artistic Exploration, and Critical Prompt Engineering. arXiv.
34. Kaji’c, I., Wiles, O., Albuquerque, I., Bauer, M., Wang, S., Pont-Tuset, J., and Nematzadeh, A. (2024). Evaluating Numerical Reasoning in Text-to-Image Models. arXiv.
35. Liu, Z., Liang, W., Zhao, Y., Chen, B., Liang, L., Wang, L., Li, J., and Yuan, Y. (2024). Glyph-ByT5-v2: A Strong Aesthetic Baseline for Accurate Multilingual Visual Text Rendering. arXiv.
36. Zhao, Y. (2023). UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models. arXiv.10.1007/978-3-031-72751-1_13 ↗
37. Yang, Y., Gui, D., Yuan, Y., Liang, W., Ding, H., Hu, H., and Chen, K. (2023, January 10–16). GlyphControl: Glyph Conditional Control for Visual Text Generation. Proceedings of the Advances in Neural Information Processing Systems 36 (NeurIPS 2023), New Orleans, LA, USA.
38. Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model
39. Chen, J., Huang, Y., Lv, T., Cui, L., Chen, Q., and Wei, F. (2023, January 10–16). TextDiffuser: Diffusion Models as Text Painters. Proceedings of the Advances in Neural Information Processing Systems 36 (NeurIPS 2023), New Orleans, LA, USA.
40. Lakhanpal, S., Chopra, S., Jain, V., Chadha, A., and Luo, M. (2024). Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation. arXiv.
41. Li, W., Li, G., Lan, Z., Xu, X., Zhuang, W., Liu, J., Xiao, X., and Su, J. (2024). Empowering Backbone Models for Visual Text Generation with Input Granularity Control and Glyph-Aware Training. arXiv.10.18653/v1/2024.emnlp-main.455 ↗
42. Tuo, Y., Geng, Y., and Bo, L. (2024). AnyText2: Visual Text Generation and Editing with Customizable Attributes. arXiv.