Benchmarking the ancient books capability of multimodal large language models
Source abstract
Abstract
Although evaluation benchmarks for general multimodal large language models (MLLMs) are increasingly prevalent, the systematic evaluation of their capabilities for processing ancient texts remains underdeveloped. Ancient books, as cultural heritage artifacts, integrate rich textual and visual elements. Due to their unique cross-linguistic complexity and multimodal composition, they pose challenges to the evaluation of the multifaceted capabilities of MLLMs. To address this issue, we propose benchmarking the ancient book capabilities of MLLMs (BABMLLM), a specialized benchmark designed to evaluate their performance specifically within the domain of ancient books. This benchmark comprises seven curated datasets, enabling comprehensive evaluation across four core tasks relevant to ancient book processing: ancient book translation, text recognition, image captioning, and image-text consistency judgment. Furthermore, BABMLLM provides a standardized reference for evaluating MLLMs in the context of ancient books and establishes a foundation for selecting suitable base models for subsequent domain-specific development.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (37)
Show source-provided references
1. Varnum, M. E. W. et al. Large Language Models based on historical text could offer informative tools for behavioral science. Proc. Natl. Acad. Sci. USA 121, e2407639121 (2024).10.1073/pnas.2407639121 ↗
2. Cao, J. et al. TongGu: Mastering Classical Chinese Understanding with Knowledge-Grounded Large Language Models. arXiv preprint arXiv:2407.03937, https://arxiv.org/abs/2407.03937 (2024).10.18653/v1/2024.findings-emnlp.243 ↗
3. Ehrmann, M. et al. Named entity recognition and classification in historical documents: a survey. ACM Comput. Surv. 56, 1–47 (2023).10.1145/3604931 ↗
4. Yang, K. et al. An automatic evaluation metric for Ancient-Modern Chinese translation. Neural Comput. Appl. 33, 3855–3867 (2021).10.1007/s00521-020-05216-8 ↗
5. Fan, T. & Wang, H. Research of Chinese intangible cultural heritage knowledge graph construction and attribute value extraction with graph attention network. Inf. Process. Manag. 59, 102753 (2022).10.1016/j.ipm.2021.102753 ↗
6. Smits, T. & Wevers, M. A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections. Digit. Scholarsh. Humanit. 38, 1267–1280 (2023).10.1093/llc/fqad008 ↗
7. Chen, C. Construction of ancient literature shared resource database based on image scanning and recognition (OCR) technology[C]//International Conference on Innovative Computing. 212-223. https://link.springer.com/chapter/10.1007/978-981-97-4193-9_23, (Springer Nature Singapore, 2024).10.1007/978-981-97-4193-9_23 ↗
8. Li, B. et al. Overview of EvaHan2024: The First International Evaluation on Ancient Chinese Sentence Segmentation and Punctuation[C]//Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA)@ LREC-COLING-2024. 229-236. https://aclanthology.org/2024.lt4hala-1.27/ (2024).
9. Song, X. Y., Zhou, J. & Liu, W. Study on the construction of intelligent machine translation model for TCM ancient books. Chin. J. Libr. Inf. Sci. Tradit. Chin. Med. 48, 130–135 (2024).
10. Liu, J. F., Ma, X. L., Wang, L. Y. & Pei, L. How can generative artificial intelligence techniques facilitate intelligent research into ancient books? Comput. Cult. Herit. https://doi.org/10.1145/3690391, https://dl.acm.org/doi/abs/10.1145/3690391 (2024).10.1145/3690391 ↗
11. Zhang, X. B. & Zhang, P. Research on the challenges and development paths of digital publishing of ancient books—taking “Shi-dian Ancient Books” as an example. J. Beijing Inst. Graph. Commun. 32, 1–6 (2024).
12. Fetaya, E. et al. Restoration of fragmentary Babylonian texts using recurrent neural networks. Proc. Natl. Acad. Sci. USA 117, 22743–22751 (2020).10.1073/pnas.2003794117 ↗
13. Hanif, M. et al. Restoration and content analysis of ancient manuscripts via color space-based segmentation. Plos One 18, e0282142 (2023).10.1371/journal.pone.0282142 ↗
14. Tong, P., Long, B. X. & Yong, C. Research on Tibetan Ujin ancient book character recognition based on attention mechanism. Comput. Technol. Dev. 33, 163–168 (2023).
15. Ding, J. S. et al. MileBench: Benchmarking MLLMs in Long Context[C]//First Conference on Language Modeling. https://openreview.net/forum?id=Uhwze2LEwq (2024).
16. Wang, H. et al. Multimodal needle in a haystack: benchmarking long-context capability of multimodal large language models. arXiv preprint arXiv:2406.11230, https://arxiv.org/abs/2305.07895 (2024).
17. Liu, Y. et al. OCRBench: on the hidden mystery of OCR in large multimodal models. Sci. China Inf. Sci. 67, 220102 (2024).10.1007/s11432-024-4235-6 ↗
18. Tong, S. et al. Eyes wide shut? Exploring the visual shortcomings of multimodal LLMS[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9568-9578. https://openaccess.thecvf.com/content/CVPR2024/html/Tong_Eyes_Wide_Shut_Exploring_the_Visual_Shortcomings_of_Multimodal_LLMs_CVPR_2024_paper.html (2024).10.1109/CVPR52733.2024.00914 ↗
19. Wang, T. et al. Equivariant similarity for vision-language foundation models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 11998-12008. https://openaccess.thecvf.com/content/ICCV2023/html/Wang_Equivariant_Similarity_for_Vision-Language_Foundation_Models_ICCV_2023_paper.html (2023).10.1109/ICCV51070.2023.01102 ↗
20. Wang, P. et al. Fvqa: Fact-based visual question answering. IEEE Trans. pattern Anal. Mach. Intell. 40, 2413–2427 (2017).10.1109/TPAMI.2017.2754246 ↗
21. Marino, K. et al. Ok-vqa: A visual question answering benchmark requiring external knowledge[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.: 3195-3204. https://openaccess.thecvf.com/content_CVPR_2019/html/Marino_OK-VQA_A_Visual_Question_Answering_Benchmark_Requiring_External_Knowledge_CVPR_2019_paper.html (2019).10.1109/CVPR.2019.00331 ↗
22. Schwenk, D. et al. A-okvqa: A benchmark for visual question answering using world knowledge[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 146-162. https://link.springer.com/chapter/10.1007/978-3-031-20074-8_9 (2022).10.1007/978-3-031-20074-8_9 ↗
23. Zhu, C. et al. Scanreason: Empowering 3d visual grounding with reasoning capabilities[C]. ECCV, https://link.springer.com/chapter/10.1007/978-3-031-73242-3_9 (2024).10.1007/978-3-031-73242-3_9 ↗
24. Chen, B. et al. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.: 14455-14465. https://openaccess.thecvf.com/content/CVPR2024/html/Chen_SpatialVLM_Endowing_Vision-Language_Models_with_Spatial_Reasoning_Capabilities_CVPR_2024_paper.html (2024).10.1109/CVPR52733.2024.01370 ↗
25. Li, J. et al. Fine-tuning multimodal LLMS to follow zero-shot demonstrative instructions[C]. The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=BXY6fe7q31&utm_source=ainews&utm_medium=email&utm_campaign=ainews-google-solves-text-to-video (2023).
26. Liu, C. et al. Intelligent grimm-open-ended visual storytelling via latent diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.: 6190-6200. https://openaccess.thecvf.com/content/CVPR2024/html/Liu_Intelligent_Grimm_-_Open-ended_Visual_Storytelling_via_Latent_Diffusion_Models_CVPR_2024_paper.html (2024).10.1109/CVPR52733.2024.00592 ↗
27. Chen, C. et al. CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model. arXiv preprint arXiv:2403.08350 https://arxiv.org/abs/2403.08350 (2024).
28. Ding, P. et al. Hallu-pi: Evaluating hallucination in multi-modal large language models within perturbed inputs[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 10707-10715. https://dl.acm.org/doi/abs/10.1145/3664647.3681251 (2024).10.1145/3664647.3681251 ↗
29. Wang, W. et al. Asclepius: a spectrum evaluation benchmark for medical multi-modal large language models. arXiv preprint arXiv:2402.11217, https://arxiv.org/abs/2402.11217 (2024).
30. Bai, F. et al. M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578 https://arxiv.org/abs/2404.00578 (2024).
31. Sermanet, P. et al. Robovqa: multimodal long-horizon reasoning for robotics[C]//2024 IEEE International Conference on Robotics and Automation (ICRA). 645–652. https://ieeexplore.ieee.org/abstract/document/10610216/ (IEEE, 2024).10.1109/ICRA57147.2024.10610216 ↗
32. Qian, T. et al. Nuscenes-qa: a multi-modal visual question answering benchmark for autonomous driving scenario. Proc. AAAI Conf. Artif. Intell. 38, 4542–4550 (2024).
33. Anderson, L. W. & Krathwohl, D. R. A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives: complete edition[M]. Addison Wesley Longman, Inc., https://eduq.info/xmlui/handle/11515/18824 (2001).
34. Lu, J. et al. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in Neural Information Processing Systems, 32, https://proceedings.neurips.cc/paper_files/paper/2019/hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html (2019).
35. Papineni, K. et al. Bleu: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318. https://aclanthology.org/P02-1040.Pdf (2002).10.3115/1073083.1073135 ↗
36. Popović, M. chrF: character n-gram F-score for automatic MT evaluation[C]//Proceedings of the tenth workshop on statistical machine translation. 392–395. https://aclanthology.org/W15-3049.pdf (2015).10.18653/v1/W15-3049 ↗
37. Lin, C. Y. Rouge: A package for automatic evaluation of summaries[C]//Text summarization branches out. 74–81. https://aclanthology.org/W04-1013.pdf (2004).