Multilingual Intelligent Retrieval System via Unified End-to-End OCR and Hybrid Search
Source abstract
Abstract
This study addresses the limitations of current Optical Character Recognition (OCR) systems in supporting minority languages and integrating intelligent retrieval functions. We propose an integrated system that combines an advanced end-to-end OCR model with a novel hybrid search approach. First, we developed the MultiLang-OCR-30K dataset containing 30,000 annotated samples of handwritten Chinese, Tibetan, and Uyghur texts. Second, we extended the GOT model using a freeze encoder–fine-tune decoder strategy to enhance multilingual capabilities. Finally, we designed a character-level hybrid retrieval framework integrating TF-IDF efficiency with Sentence-BERT semantic strength. Experimental results show our extended GOT model achieves sentence accuracies of 82.3%, 76.5%, and 78.1% for handwritten Chinese, Tibetan, and Uyghur, respectively. The hybrid search improves F1 score by 28.7% over TF-IDF alone while maintaining 23 ms average response time. This system provides a practical solution for multilingual document digitization and management, thereby bridging the technological gap for minority languages.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques · Topic Modeling · Image Retrieval and Classification Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Xinjiang Normal University, Urumqi 830017, China
Xinjiang Engineering Research Center for Smart Education and Applications, Urumqi 830054, China
Library of Xinjiang Normal University, Xinjiang Normal University, Urumqi 830017, China
Funding
National Natural Science Foundation of China · Doctoral Research Foundation of Xinjiang Normal University · Xinjiang Normal University Youth Top Talents Project · General Program of the Natural Science Foundation of Xinjiang Uygur Autonomous Region · General Program of the Science and Technology Program of Xinjiang Uygur Autonomous Region · Xinjiang Key Research and Development Program
References (38)
Show source-provided references
1. Wei, H., Liu, C., Chen, J., Wang, J., Kong, L., Xu, Y., Ge, Z., Zhao, L., Sun, J., and Peng, Y. (2024). General OCR Theory: Towards OCR-2.0 via a Unified End-to-End Model. arXiv.
2. Tuo, Y., Xiang, W., He, J.Y., Geng, Y., and Xie, X. (2023). AnyText: Multilingual Visual Text Generation and Editing. arXiv.
3. Smith, R. (2007, January 23–26). An Overview of the Tesseract OCR Engine. Proceedings of the Ninth International Conference on Document Analysis and Recognition, Curitiba, Brazil.
4. Du, Y., Li, C., Guo, R., Cui, C., Liu, W., Zhou, J., Lu, B., Yang, Y., Liu, Q., and Hu, X. (2021). PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR System. arXiv.
5. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models from Natural Language Supervision. Proceedings of the International Conference on Machine Learning, Virtual. Available online: https://proceedings.mlr.press/v139/radford21a.html.
6. Attention is All You Need
7. TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
8. Analysis of Optical Character Recognition Using EasyOCR under Image Degradation10.1088/1742-6596/2641/1/012001 ↗
9. Ma, J., Zhao, M., Chen, C., Wang, R., Niu, D., Lu, H., and Lin, X. (2023). GlyphDraw: Learning to Draw Chinese Characters in Image Synthesis Models Coherently. arXiv.
10. TextDiffuser: Diffusion Models as Text Painters
11. GlyphControl: Glyph Conditional Control for Visual Text Generation
12. A Survey on Image Data Augmentation for Deep Learning10.1186/s40537-019-0197-0 ↗
13. Using TF-IDF to Determine Word Relevance in Document Queries
14. The Probabilistic Relevance Framework: BM25 and Beyond10.1561/1500000019 ↗
15. Reimers, N., and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. arXiv.10.18653/v1/D19-1410 ↗
16. Karpukhin, V., Oguz, B., Min, S., Lewis, P.S.H., Wu, L., Edunov, S., Chen, D., and Yih, W.T. (2020, January 16–20). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.550 ↗
17. Cross-lingual Language Model Pretraining
18. Chi, Z., Huang, S., Dong, L., Ma, S., Zheng, B., Singhal, S., Bajaj, P., Song, X., Mao, X.L., and Huang, H.Y. (2022, January 22–27). XLM-E: Cross-lingual Language Model Pre-training via ELECTRA. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland. Available online: https://aclanthology.org/2022.acl-long.427/.10.18653/v1/2022.acl-long.427 ↗
19. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2021, January 6–11). mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.41 ↗
20. Yang, Y., Cer, D., Ahmad, A., Guo, M., Law, J., Constant, N., Abrego, G.H., Yuan, S., Tar, C., and Sung, Y.H. (2022, January 22–27). Multilingual Universal Sentence Encoder for Semantic Retrieval. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Dublin, Ireland.
21. Koilada, D.K. (2026, January 06). Hybrid Semantic Retrieval: Augmenting Weighted TF-IDF with BERT for Enhanced Question Answering, Eng. Arch., Available online: https://www.atlantis-press.com/proceedings/icsiaiml-25/126021198.10.31224/5341 ↗
22. Billion-scale Similarity Search with GPUs10.1109/TBDATA.2019.2921572 ↗
23. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2022, January 22–27). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland.
24. Kingma, D.P., and Welling, M. (2013). Auto-encoding Variational Bayes. arXiv.
25. Overcoming Catastrophic Forgetting in Neural Networks10.1073/pnas.1611835114 ↗
26. Zhuang, L., Wayne, L., Ya, S., and Jun, Z. (2021, January 13–15). A Robustly Optimized BERT Pre-training Approach with Post-Training. Proceedings of the 20th Chinese National Conference on Computational Linguistics, Huhhot, China. Available online: https://aclanthology.org/2021.ccl-1.108.
27. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 6). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
28. Understanding the difficulty of training deep feedforward neural networks
29. Loshchilov, I., and Hutter, F. (2017). Decoupled Weight Decay Regularization. arXiv.
30. Loshchilov, I., and Hutter, F. (2016). SGDR: Stochastic Gradient Descent with Warm Restarts. arXiv.
31. Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., and Zhang, J. (2023, January 6–10). UReader: Universal OCR-Free Visually-Situated Language Understanding with Multimodal Large Language Model. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore. Available online: https://arxiv.org/abs/2310.05126.10.18653/v1/2023.findings-emnlp.187 ↗
32. Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yang, J., Sun, J., Han, C., and Zhang, X. (October, January 29). Vary: Scaling Up the Vision Vocabulary for Large Vision-Language Model. Proceedings of the Computer Vision—ECCV 2024 18th European Conference, Milan, Italy.10.1007/978-3-031-73235-5_23 ↗
33. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., and Huang, F. (2023). Qwen Technical Report. arXiv.
34. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
35. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.
36. Li, Y., Mao, H., Girshick, R., and He, K. (2022, January 23–27). Exploring Plain Vision Transformer Backbones for Object Detection. Proceedings of the European Conference on Computer Vision (ECCV 2022), Tel Aviv, Israel.10.1007/978-3-031-20077-9_17 ↗
37. Verifiable Coded Computation of Multiple Functions
38. Zhou, X., and Shroff, N. (2025). Cost-Efficient Distributed Learning via Combinatorial Multi-Armed Bandits. Entropy, 27.10.3390/e27050541 ↗