Multimodal context-aware translation of the endangered dongba script
Source abstract
Abstract
Dongba script is one of the few pictographic writing systems still in use, characterized by nonlinear structure and strong integration of text and images, which hinders direct application of sequence-based machine translation. We make the first attempt to introduce an open-source multimodal large language model (MLLM) for paragraph-level Dongba-to-Chinese translation. To handle its nonlinear semantics, we propose a structured contextual semantic augmentation (SCSA) method, which segments Dongba images into coherent units and generates diverse training samples through reordering and spatial splicing. This guides the model to capture independent semantic units and their combinatorial relations. To support this task, we construct DongbaMMTCorpus, the first large-scale Dongba-Chinese image-text parallel corpus. Experiments indicate that SCSA can improve the ability of MLLMs to process nonlinear structures and enhance translation quality, providing a preliminary step toward automated translation of pictographic scripts.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Natural Language Processing Techniques · Handwritten Text Recognition Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (37)
Show source-provided references
1. Xu, D. Digital approaches to understanding Dongba pictographs. Int. J. Digit. Humanit. 4, 131–146 (2023).10.1007/s42803-022-00057-4 ↗
2. Wikipedia contributors. Dongba symbols. https://en.wikipedia.org/wiki/Dongba_symbols (2025).
3. United Nations Educational, Scientific and Cultural Organization. Ancient Naxi Dongba literature manuscripts - memory of the world. https://www.unesco.org/en/memory-world/ancient-naxi-dongba-literature-manuscripts (2003).
4. Fiorucci, M. et al. Machine learning for cultural heritage: a survey. Pattern Recognit. Lett. 133, 102–108 (2020).10.1016/j.patrec.2020.02.017 ↗
5. Bi, X., Sun, Z. & Chen, Z. A novel unsupervised contrastive learning framework for ancient Yi script character dataset construction. npj Herit. Sci. 13, 39 (2025).10.1038/s40494-025-01600-6 ↗
6. Bi, X., Han, L. & Qiao, W. ShuiAttNet: fully convolutional attention network for Shuishu character recognition. Expert Syst. Appl. 282, 127613 (2025).10.1016/j.eswa.2025.127613 ↗
7. Bi, X. & Luo, Y. Incomplete handwritten Dongba character image recognition by multiscale feature restoration. Herit. Sci. 12, 218 (2024).10.1186/s40494-024-01329-8 ↗
8. Ma, Y. et al. STEF: a Swin transformer-based enhanced feature pyramid fusion model for Dongba character detection. Herit. Sci. 12, 206 (2024).10.1186/s40494-024-01321-2 ↗
9. Bi, X. et al. DongbaMIE: a multimodal information extraction dataset for evaluating semantic understanding of Dongba pictograms. In Findings of the Association for Computational Linguistics. (EMNLP, 2025).
10. Ma, X., Lan, M., Hu, W. & Lu, Y. Dongba machine translation with transfer Learning: leveraging pre-trained ancient Chinese models. ACM Trans. Asian Low. Resour. Lang. Inf. Process. 24, 1–17 (2025).10.1145/3721980 ↗
11. Gao, S., Yang, X., Yu, Z., Pan, X. & Guo, J. Chinese-Naxi machine translation method based on Naxi dependency language model. Int. J. Mach. Learn. Cybern. 8, 333–342 (2017).10.1007/s13042-014-0325-2 ↗
12. Gao, S., Tang, Z., Yu, Z., Liu, C. & Wu, L. Chinese-Naxi syntactic statistical machine translation based on tree-to-tree. Int. J. Inf. Commun. Technol. 13, 351–360 (2018).
13. Gao, S., Yu, Z., Liu, C., Chen, L. & Hong, X. A Chinese-Naxi tree-to-tree machine translation method based on subtree alignment. Math. Probl. Eng. 2015, 414963 (2015).10.1155/2015/414963 ↗
14. Poupard, D. Revitalising Naxi dongba as a ‘pictographic’ vernacular script. J. Chin. Writ. Syst. 3, 53–67 (2019).10.1177/2513850218814405 ↗
15. Tan, Z. et al. Neural machine translation: a review of methods, resources, and tools. AI Open 1, 5–21 (2020).10.1016/j.aiopen.2020.11.001 ↗
16. Li, S., Bi, X., Liu, T. & Chen, Z. Information dropping data augmentation for machine translation quality estimation. IEEE/ACM Trans. Audio Speech Lang. Process. 32, 2112–2124 (2024).10.1109/TASLP.2024.3380996 ↗
17. Ranathunga, S. et al. Neural machine translation for low-resource languages: a survey. ACM Comput. Surv. 55, 1–37 (2023).10.1145/3567592 ↗
18. Sulubacak, U. et al. Multimodal machine translation through visuals and speech. Mach. Transl. 34, 97–147 (2020).10.1007/s10590-020-09250-0 ↗
19. Zhang, Z. et al. Understand layout and translate text: unified feature-conductive end-to-end document image translation. IEEE Trans. Pattern Anal. Mach. Intell. 47, 3358–3376 (2025).10.1109/TPAMI.2025.3530998 ↗
20. Ghandi, T., Pourreza, H. & Mahyar, H. Deep learning approaches on image captioning: a review. ACM Comput. Surv. 56, 1–39 (2023).10.1145/3617592 ↗
21. Nguyen, T., Gadre, S. Y., Ilharco, G., Oh, S. & Schmidt, L. Improving multimodal datasets with image captioning. Adv. Neural Inf. Process. Syst. 36, 22047–22069 (2023).
22. Li, B. et al. On vision features in multimodal machine translation. Proc. 60th Annu. Meet. Assoc. Comput. Linguist. 1, 6327–6337 (2022).
23. Guo, J., Hou, Z., Xian, Y. & Yu, Z. Progressive modality-complement aggregative multitransformer for domain multi-modal neural machine translation. Pattern Recognit. 149, 110294 (2024).10.1016/j.patcog.2024.110294 ↗
24. Stefanini, M. et al. From show to tell: a survey on deep learning-based image captioning. IEEE Trans. Pattern Anal. Mach. Intell. 45, 539–559 (2023).10.1109/TPAMI.2022.3148210 ↗
25. Bai, S. et al. Qwen2.5-VL. Technical report. https://arxiv.org/abs/2502.13923 (2025).
26. Hong, W. et al. CogVLM2: Visual language models for image and video understanding. https://arxiv.org/abs/2408.16500 (2024).
27. Yao, Y. et al. Efficient GPT-4V level multimodal large language model for deployment on edge devices. Nat. Commun. 16, 5509 (2025).10.1038/s41467-025-61040-5 ↗
28. Liu, H. et al. LLaVA-NeXT: improved reasoning, OCR, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/ (2024).
29. Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting of the Association for Computational Linguistics 311–318 (Association for Computational Linguistics, 2002).10.3115/1073083.1073135 ↗
30. Ma, Y. et al. Dataset for Single Character Detection in Dongba Manuscripts. Sci. Data 12, 1075 (2025).10.1038/s41597-025-05434-6 ↗
31. Luo, Y., Sun, Y. & Bi, X. Multiple attentional aggregation network for handwritten Dongba character recognition. Expert Syst. Appl. 213, 118865 (2023).10.1016/j.eswa.2022.118865 ↗
32. Xing, J., Bi, X. & Weng, Y. A multi-scale hybrid attention network for sentence segmentation line detection in Dongba scripture. Mathematics 11, 1–18 (2023).10.3390/math11153392 ↗
33. He, W. & He, J. An Annotated Collection of Naxi Dongba Manuscripts (Yunnan People’s Publishing House, 1999).
34. Zheng, Y., Zhang, R., Zhang, J., Ye, Y. & Luo, Z. LlamaFactory: unified efficient fine-tuning of 100+ language models. Proc. 62nd Annu. Meet. Assoc. Comput. Linguist. 3, 400–410 (2024).
35. Loshchilov, I. & Hutter, F. Decoupled Weight Decay Regularization. International Conference on Learning Representations. https://openreview.net/forum?id=Bkg6RiCqY7 (2019).
36. Hu, E. J. et al. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. https://openreview.net/forum?id=nZeVKeeFYf9 (2022).
37. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr (2020).