Synthetized Multilanguage OCR Using CRNN and SVTR Models for Realtime Collaborative Tools
Source abstract
Abstract
Background: Remote diagnosis using collaborative tools have led to multilingual joint working sessions in various domains, including comprehensive health care, and resulting in more inclusive health care services. One of the main challenges is providing a real-time solution for shared documents and presentations on display to improve the efficacy of noninvasive, safe, and far-reaching collaborative models. Classic optical character recognition (OCR) solutions fail when there is a mixture of languages or dialects or in case of the participation of different technical levels and skills. Due to the risk of misunderstandings caused by mistranslations or lack of domain knowledge of the interpreters involved, the technological pipeline also needs artificial intelligence (AI)-supported improvements on the OCR side. This study examines the feasibility of machine learning-supported OCR in a multilingual environment. The novelty of our method is that it provides a solution not only for different speaking languages but also for a mixture of technological languages, using artificially created vocabulary and a custom training data generation approach. Methods: A novel hybrid language vocabulary creation method is utilized in the OCR training process in combination with convolutional recurrent neural networks (CRNNs) and a single visual model for scene text recognition within the patch-wise image tokenization framework (SVTR). Data: In the research, we used a dedicated Python-based data generator built on dedicated collaborative tool-based templates to cover and simulated the real-life variances of remote diagnosis and co-working collaborative sessions with high accuracy. The generated training datasets ranged from 66 k to 8.5 M in size. Twenty-one research results were analyzed. Instruments: Training was conducted by using tuned PaddleOCR with CRNN and SVTR modeling and a domain-specific, customized vocabulary. The Weight & Biases (WANDB) machine learning (ML) platform is used for experiment tracking, dataset versioning, and model evaluation. Based on the evaluations, the training dataset was adjusted by using a different language corpus or/and modifications applied to templates. Results: The machine learning models recognized the multilanguage/hybrid texts with high accuracy. The highest precision scores achieved are 90.25%, 91.35%, and 93.89%. Conclusions: machine learning models for special multilanguages, including languages with artificially made vocabulary, perform consistently with high accuracy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques · Image Retrieval and Classification Techniques · Medical Imaging and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical Engineering and Information Technology, George Emil Palade University of Medicine, Pharmacy, Science, and Technology of Targu Mures, Str. Nicolae Iorga, Nr. 1, 540088 Targu Mures, Romania
Department of Physiotherapy, University of Malaga, 29071 Malaga, Spain
Biomedical Research Institute of Malaga (IBIMA), 29590 Malaga, Spain
Faculty of Health Science, School of Clinical Science, Queensland University Technology, Brisbane 4000, Australia
Legal and Forensic Medicine Area, Department of Human Anatomy, Legal Medicine and History of Science, Faculty of Medicine, University of Malaga, 29071 Malaga, Spain
Computational Intelligence Research Group, Sapientia Hungarian University of Transylvania, 540485 Targu Mures, Romania
Physiological Controls Research Center, Óbuda University, 1034 Budapest, Hungary
Funding
ITware, Hungary · University of Malaga · Consolidator Excellence Researcher Program of Óbuda University, Budapest Hungary · Sapientia Institute for Research Programs, Romania
References (45)
Show source-provided references
1. The factors affecting team effectiveness in hospitals: The mediating role of using electronic collaborative tools10.1016/j.xjep.2021.100449 ↗
2. Biró, A., Jánosi-Rancz, K.T., Szilágyi, L., Cuesta-Vargas, A.I., Martín-Martín, J., and Szilágyi, S.M. (2022). Visual Object Detection with DETR to Support Video-Diagnosis Using Conference Tools. Appl. Sci., 12.10.3390/app12125977 ↗
3. Huang, J., Pang, G., Kovvuri, R., Toh, M., Liang, K.J., Krishnan, P., Yin, X., and Hassner, T. (2021, January 20–25). A Multiplexed Network for End-to-End, Multilingual OCR. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00452 ↗
4. An End-to-End OCR Text Reorganization Sequence Learning for Rich-text Detail Image Comprehension. European Conference on Computer Vision
5. Du, Y.N., Li, C.X., Guo, R.Y., Yin, X.T., Liu, W.W., Zhou, J., Bai, Y.F., Yu, Z.L., Yang, Y.H., and Dang, Q.Q. (2020). PP-OCR: A Practical Ultra Lightweight OCR System. arXiv.
6. Du, Y.N., Li, C.X., Guo, R.Y., Cui, C., Liu, W.W., Zhou, J., Lu, B., Yang, Y.H., Liu, Q., and Hu, W. (2021). PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR System. arXiv.
7. Nguyen, T.T.H., Jatowt, A., Coustaty, M., and Doucet, A. (2021). Survey of Post-OCR Processing Approaches. ACM Comput. Surv., 6.10.1145/3453476 ↗
8. Zhao, Z.P., Zhao, Y.Q., Bao, Z.T., Wang, H.S., Zhang, Z.X., and Li, C. (2018, January 26). Deep Spectrum Feature Representations for Speech Emotion Recognition. Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and first Multi-Modal Affective Computing of Large-Scale Multimedia Data, Seoul, Republic of Korea.10.1145/3267935.3267948 ↗
9. Fischer-Suárez, N., Lozano-Paniagua, D., García-González, J., Castro-Luna, G., Requena-Mullor, M., Alarcón-Rodríguez, R., Parrón-Carreño, T., and Nievas-Soriano, B.J. (2022). Use of Digital Technology as a Collaborative Tool among Nursing Students—Survey Study and Validation. Int. J. Environ. Res. Public Health, 19.10.3390/ijerph192114267 ↗
10. Li, X., Zhang, Y., Yuan, W., and Luo, J. (2022). Incorporating External Knowledge Reasoning for Vision-and-Language Navigation with Assistant’s Help. Appl. Sci., 12.10.3390/app12147053 ↗
11. Integrating machine translation into translator training: Towards ‘Human Translator Competence’?10.29228/transLogos.11 ↗
12. Bizzoni, Y., Juzek, T.S., España-Bonet, C., Chowdhury, K.D., van Genabith, J., and Teich, E. (2020). Proceedings of the 17th International Conference on Spoken Language Translation, Association for Computational Linguistics.
13. Zhang, B., Bapna, A., Sennrich, R., and Firat, O. (2021, January 3–7). Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation. Proceedings of the International Conference on Learning Representations, Virtual. Available online: https://openreview.net/pdf?id=Wj4ODo0uyCF.
14. Optical Character Recognition and Neural Machine Translation Using Deep Learning Techniques10.1007/978-981-33-4543-0_30 ↗
15. Yang, J., Yin, Y.W., Ma, S.M., Zhang, D.D., Li, Z.J., and Wei, F.R. (2022). High-resource Language-specific Training for Multilingual Neural Machine Translation. Int. Jt. Conf. Artif. Intell., 4436–4442.10.24963/ijcai.2022/619 ↗
16. Qi, J.W., and Peng, Y.X. (2018). Cross-modal bidirectional translation via reinforcement learning. Int. Jt. Conf. Artif. Intell., 2630–2636.10.24963/ijcai.2018/365 ↗
17. Towards modeling user behavior in interactions mediated through an automated bidirectional speech translation system10.1016/j.csl.2009.04.008 ↗
18. Ding, L.A., Wu, D., and Tao, D.C. Improving neural machine translation by bidirectional training. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.10.18653/v1/2021.emnlp-main.263 ↗
19. Improving the accuracy of tesseract OCR engine for machine printed Hindi documents10.1063/5.0101164 ↗
20. Rijhwani, S., Anastasopoulos, A., and Neubig, G. (2020, January 16–20). OCR Post Correction for Endangered Language Texts Pages. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), online. Available online: https://aclanthology.org/2020.emnlp-main.478.pdf.10.18653/v1/2020.emnlp-main.478 ↗
21. Gunna, S., Saluja, R., and Jawahar, C.V. (2022). Improving Scene Text Recognition for Indian Languages with Transfer Learning and Font Diversity. J. Imaging, 8.10.3390/jimaging8040086 ↗
22. Ignat, O., Maillard, J., Chaudhary, V., and Guzmán, F. (2022). OCR Improves Machine Translation for Low-Resource Languages Pages. arXiv.10.18653/v1/2022.findings-acl.92 ↗
23. Multi-Lingual Optical Character Recognition System Using the Reinforcement Learning of Character Segmenter10.1109/ACCESS.2020.3025769 ↗
24. AI-backed OCR in Healthcare10.1016/j.procs.2022.09.169 ↗
25. Bartz, C., Yang, H.J., and Meinel, C. (2017). STN-OCR: A single Neural Network for Text Detection and Text Recognition. arXiv.
26. Hyperspectral image analysis techniques for the detection and classification of the early onset of plant disease and stress10.1186/s13007-017-0233-z ↗
27. (2023, March 22). PaddleOCR. Available online: https://github.com/PaddlePaddle/PaddleOCR.
28. (2023, March 22). Paddle Japanese Model—Japan Ultra-Lightweight OCR Model. Available online: https://github.com/1849349137/PaddleOCR.
29. Wu, H., and Prasad, S. (2017). Convolutional Recurrent Neural Networks for Hyperspectral Data Classification. Remote Sens., 9.10.3390/rs9030298 ↗
30. Du, Y.K., Chen, Z.N., Jia, C.Y., Yin, X.T., Zheng, T.L., Li, C.X., Du, Y.N., and Jiang, Y.G. (, January 23–29). SVTR: Scene Text Recognition with a Single Visual Model. Proceedings of the 31st International Joint Conference on Artificial Intelligence Main Track, Vienna, Austria.
31. Kloft, M., Stiehler, F., Zheng, Z.L., and Pinkwart, N. (2014). EMNLP Workshop on Analysis of Large Scale Social Interaction in MOOCs, Association for Computational Linguistics.
32. Efficient CRNN Recognition Approaches for Defective Characters in Images10.12785/ijcds/1201114 ↗
33. Kang, P., and Singh, A.K. (2023, March 22). CTC—Problem Statement. The AI Learner. Available online: https://theailearner.com/.
34. An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition10.1109/TPAMI.2016.2646371 ↗
35. Keren, G., and Schuller, B. (2016, January 24–29). Convolutional RNN: An Enhanced Model for Extracting Features from Sequential Data. Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada. Available online: https://arxiv.org/pdf/1602.05875.pdf.10.1109/IJCNN.2016.7727636 ↗
36. Semi-Supervised Deep Learning Using Pseudo Labels for Hyperspectral Image Classification10.1109/TIP.2017.2772836 ↗
37. Gan, Z., Singh, P.D., Joshi, A., He, X.D., Chen, J.S., Gao, J.F., and Deng, L. (2017). Character-level Deep Conflation for Business Data Analytics. arXiv.10.1109/ICASSP.2017.7952551 ↗
38. EMD and LSTM Hybrid Deep Learning Model for Predicting Sunspot Number Time Series with a Cyclic Pattern10.1007/s11207-020-01653-9 ↗
39. Zhuang, J., Ren, Y., Li, X., and Liang, Z. (2022, January 27–28). Text-Level Contrastive Learning for Scene Text Recognition. Proceedings of the 2022 International Conference on Asian Language Processing (IALP), Singapore.10.1109/IALP57159.2022.9961322 ↗
40. Adaptive detrending to accelerate convolutional gated recurrent unit training for contextual video recognition10.1016/j.neunet.2018.05.009 ↗
41. Brownlee, J. (2023, March 22). Understand the Impact of Learning Rate on Neural Network Performance, Deep. Learn. Perform., Available online: https://machinelearningmastery.com/understand-the-dynamics-of-learning-rate-on-deep-learning-neural-networks.
42. Rerunning OCR: A Machine Learning Approach to Quality Assessment and Enhancement Prediction
43. Almuhaideb, A.M., Aslam, N., Alabdullatif, A., Altamimi, S., Alothman, S., Alhussain, A., Aldosari, W., Alsunaidi, S.J., and Alissa, K.A. (2022). Homoglyph Attack Detection Model Using Machine Learning and Hash Function. J. Sens. Actuator Netw., 11.10.3390/jsan11030054 ↗
44. Majumder, M.T.H., Rahman, M.M., Iqbal, A., and Rahman, M.S. (2020). Convolutional Neural Network Based Ensemble Approach for Homoglyph Recognition. Math. Comput. Appl., 25.10.3390/mca25040071 ↗
45. Suzuki, H., Chiba, D., Yoneya, Y., Mori, T., and Goto, S. (2019, January 21–23). ShamFinder: An Automated Framework for Detecting IDN Homographs. Proceedings of the IMC’19: ACM Internet Measurement Conference, Amsterdam, The Netherlands.10.1145/3355369.3355587 ↗