GAN Augmented Hybrid Transformer Network (GHTNet) For Ancient Tamil Stone Inscription Recognition
Source abstract
Abstract
Ancient Tamil inscriptions, like Brahmi and Vattezhuthu, hold immense historical and cultural value. However, the degradation of stone surfaces, occluded characters, and script evolution over centuries pose significant challenges to accurate digitization and translation. This paper proposes GHTNet, a novel GAN-augmented Hybrid Transformer Network capable of recognizing and translating Tamil stone inscriptions into modern Tamil directly on mobile devices. The proposed pipeline begins with DnCNN-based de-noising and perspective projection. To improve recognition color feature-based augmentation is performed using CycleGAN and Conditional GAN. Character recognition is achieved using a combination of Swin Transformer and TrOCR, effectively extracting complex script features. Subsequently, Decoupled Attention Network (DAN) and VisionLAN are employed for word recognition. The final stage involves sentence formation and translation using Graph Attention Networks (GAT) and Neural Machine Translation (NMT) models. Extensive experiments validate that GHTNet significantly reduces character and word error rates while achieving high translation accuracy of 98%.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques · Image Processing and 3D Reconstruction · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (66)
Show source-provided references
1. Giridharan, R., Vellingiriraj E. K. & Balasubramanie, P. Identification of Tamil ancient characters and information retrieval from temple epigraphy using image zoning. 2016 International Conference on Recent Trends in Information Technology (ICRTIT), Chennai, India, 2016, pp. 1−7, https://doi.org/10.1109/ICRTIT.2016.7569600.10.1109/ICRTIT.2016.7569600 ↗
2. Deepa, J., Arvind, B., Bharanidharan, G. & Dinesh, S. An empirical analysis on recognition of ancient tamil inscriptions using machine learning. In 2024 IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT) 1739−1744 (IEEE, 2024).10.1109/IC2PCT60090.2024.10486642 ↗
3. Sivashanth, S., Charles, E. Y. A. & Manivannan, S. A fully automated approach for enhancing ancient tamil inscriptions using deep learning. In 2024 8th SLAAI International Conference on Artificial Intelligence (SLAAI-ICAI) 1−5 (IEEE, 2024).10.1109/SLAAI-ICAI63667.2024.10844964 ↗
4. Dhivya, S. & Beulah, J. R. Ancient tamil character recognition from stone inscriptions–a theoretical analysis. In 2022 2nd Asian Conference on Innovation in Technology (ASIANCON) 1−8 (IEEE, 2022).10.1109/ASIANCON55314.2022.9908951 ↗
5. Devi, D. R., Kanna, M. D. & Rajendran, S. Ancient tamil digits recognition using convolutional neural network. In AIP Conference Proceedings, 3180 (AIP Publishing, 2024).10.1063/5.0224624 ↗
6. Manigandan, T. V. V. D. V. N. B., Vidhya, V., Dhanalakshmi, V. & Nirmala, B. (2017, August). Tamil character recognition from ancient epigraphical inscription using OCR and NLP. In 2017 international conference on energy, communication, data analytics and soft computing (ICECDS) (pp. 1008−1011).10.1109/ICECDS.2017.8389589 ↗
7. Atsumi, M., KAWANO, S., MORIOKA, T. & MENG, L. Deep learning based ancient Asian character recognition. In 2020 International Conference on Advanced Mechatronic Systems (ICAMechS) 296−301 (IEEE, 2020).10.1109/ICAMechS49982.2020.9310140 ↗
8. Daiki, T. & Satlow, M. Machine learning techniques for analyzing inscriptions from Israel. In DHQ: Digital Humanities Quarterly, 17 (2023).
9. Jayanthi, J. & Maheswari, P. U. Comparative study: enhancing legibility of ancient Indian script images from diverse stone background structures using 34 different pre-processing methods. Herit. Sci. 12, 63 (2024).10.1186/s40494-024-01169-6 ↗
10. Aswathy, A. & Maheswari, P. U. Generative innovations for paleography: enhancing character image synthesis through unconditional single image models. Herit. Sci. 12, 258 (2024).10.1186/s40494-024-01373-4 ↗
11. Bhuvaneswari, S. & Kathiravan, K. Enhancing epigraphy: a deep learning approach to recognize and analyze Tamil ancient inscriptions. Neural Comput. Appl. 36, 19839–19861 (2024).10.1007/s00521-024-10137-x ↗
12. Priya, et al. Self-adaptive hybridized lion optimization algorithm with transfer learning for ancient tamil character recognition in stone inscriptions. In IEEE Access, 39621−39634 (IEEE, 2023).10.1109/ACCESS.2023.3268545 ↗
13. Munivel, M. & Enigo, V. S. Optical character recognition for printed tamizhi documents using deep neural networks. DESIDOC J. Libr. Inf. Technol. 42, 227–233 (2022).10.14429/djlit.42.4.17742 ↗
14. Gnanasivam, P., Bharath, G., Karthikeyan, V. & Dhivya, V. Handwritten Tamil character recognition using convolutional neural network. In 2021 Sixth International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET) 84−88 (IEEE, 2021).10.1109/WiSPNET51692.2021.9419451 ↗
15. Velmathi, G., Harish, D., Shangavelan, M. & Krithikshun, M. S. Image Based Character Recognition, Documentation System To Decode Inscription From Temple. arXiv https://doi.org/10.48550/arXiv.2405.17449 (2024).10.48550/arXiv.2405.17449 ↗
16. Munivel, M. & Enigo, V. F. MLIBT: A multi-level improvised binarization technique for Tamizhi inscriptions. Expert Syst. Appl. 236, 121320 (2024).10.1016/j.eswa.2023.121320 ↗
17. Babu, B. S. et al. Temple inscriptions recognition and transliteration in devanagari script. In 2023 2nd International Conference on Vision Towards Emerging Trends in Communication and Networking Technologies (ViTECoN), 1−6 (IEEE, 2023).10.1109/ViTECoN58111.2023.10157277 ↗
18. Subramani, K. & Murugavalli, S. Recognizing ancient characters from Tamil palm leaf manuscripts using convolution based deep learning. Int. J. Recent Technol. Eng. 8, 6873–6880 (2019).
19. Magrina, M. M. Convolution neural network based ancient tamil character recognition from epigraphical inscriptions. In International Research Journal of Engineering and Technology (IRJET), 7 (2020).
20. Devi, K. D., Maheswari, P. U., Polasi, P. K., Preetha, R. & Vidhyalakshmi, M. Pattern matching model for recognition of stone inscription characters. Comput. J. 66, 554–564 (2023).10.1093/comjnl/bxab177 ↗
21. Murali, V. & Sudeep, P. V. Image denoising using DnCNN: an exploration study. In Advances in Communication Systems and Networks: Select Proceedings of ComNet 2019 847−859 (Springer Singapore, 2020).10.1007/978-981-15-3992-3_72 ↗
22. Goodfellow, I. J. et al. Generative adversarial nets. arXiv https://doi.org/10.48550/arXiv.1406.2661 (2014).10.48550/arXiv.1406.2661 ↗
23. Lan, L. et al. Generative adversarial networks and its applications in biomedical informatics. Front. Public Health 8, 164 (2020).10.3389/fpubh.2020.00164 ↗
24. Sandouka, S. B., Bazi, Y. & Alajlan, N. Transformers and generative adversarial networks for liveness detection in multitarget fingerprint sensors. Sensors 21, 699 (2021).10.3390/s21030699 ↗
25. Wang, W. et al. Improving oracle bone characters recognition via a cyclegan-based data augmentation method. In International Conference on Neural Information Processing, 88−100 (Singapore: Springer Nature, 2022).10.1007/978-981-99-1645-0_8 ↗
26. Isola, P., Zhu, J. Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1125−1134 (2017).10.1109/CVPR.2017.632 ↗
27. Li, M. et al. Trocr: Transformer-based optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, 13094−13102 (AAAI, 2023).10.1609/aaai.v37i11.26538 ↗
28. Zheng, Y., Chen, Y., Wang, X., Qi, D. & Yan, Y. Ancient chinese character recognition with improved swin-transformer and flexible data enhancement strategies. Sensors 24, 2182 (2024).10.3390/s24072182 ↗
29. Ma, Y. et al. Stef: a swin transformer-based enhanced feature pyramid fusion model for dongba character detection. Herit. Sci. 12, 206 (2024).10.1186/s40494-024-01321-2 ↗
30. Wang, T. et al. Decoupled attention network for text recognition. Proc. AAAI Conf. Artif. Intell. 34, 12216–12224 (2020).
31. Wang, Y. et al. From two to one: a new scene text recognizer with visual language modeling network. In Proceedings of the IEEE/CVF International Conference on Computer Vision,14194−14203 (IEEE, 2021).10.1109/ICCV48922.2021.01393 ↗
32. Li, J., Jian, Y. & Xiong, Y. Text classification model based on graph attention networks and adversarial training. Appl. Sci. 14, 4906 (2024).10.3390/app14114906 ↗
33. Goodfellow, I.J. et al. Generative adversarial nets. Advances in neural information processing systems, 27 (2014).
34. Tan, Z. et al. Neural machine translation: a review of methods, resources, and tools. AI Open 1, 5–21 (2020).10.1016/j.aiopen.2020.11.001 ↗
35. Shi, B. et al. Aster: An attentional scene text recognizer with flexible rectification. IEEE Trans. pattern Anal. Mach. Intell. 41, 2035–2048 (2018).10.1109/TPAMI.2018.2848939 ↗
36. Qiao, Z. et al. PIMNet: a parallel, iterative and mimicking network for scene text recognition. In Proceedings of the 29th ACM International Conference on Multimedia, 2046−2055 (ACM, 2021).10.1145/3474085.3475238 ↗
37. Moudgil, A., Singh, S., Gautam, V., Rani, S. & Shah, S. H. Handwritten devanagari manuscript characters recognition using capsnet. Int. J. Cogn. Comput. Eng. 4, 47–54 (2023).
38. Chauhan, V. K., Singh, S. & Sharma, A. HCR-Net: A deep learning based script independent handwritten character recognition network. Multimed. Tools Appl. 83, 78433–78467 (2024).10.1007/s11042-024-18655-5 ↗
39. Magrina, M. M. Recognition of ancient tamil characters from epigraphical inscriptions using raspberry pi based tesseract OCR. Int. J. Sci. Res. Comput. Sci. Eng. Inf. Technol. https://doi.org/10.32628/CSEIT217230 (2021).10.32628/CSEIT217230 ↗
40. Ghosh, J., Talukdar, A. K. & Sarma, K. K. A light-weight natural scene text detection and recognition system. Multimed. Tools Appl. 83, 6651–6683 (2024).10.1007/s11042-023-15696-0 ↗
41. Zhou, Q., Gao, J., Yuan, Y. & Wang, Q. RRTrN: A lightweight and effective backbone for scene text recognition. Expert Syst. Appl. 243, 122769 (2024).10.1016/j.eswa.2023.122769 ↗
42. Haritha, J. et al. CNN based character recognition and classification in tamil palm leaf manuscripts. In 2022 International Conference on Communication, Computing and Internet of Things (IC3IoT), 1−6 (IEEE, 2022).10.1109/IC3IOT53935.2022.9767866 ↗
43. Koç, C., Özyurt, F. & Iantovics, L. B. Survey on latest advances in natural language processing applications of generative adversarial networks. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 15, e70004 (2025).10.1002/widm.70004 ↗
44. Zhu, J. Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, 2223−2232 (IEEE, 2017).10.1109/ICCV.2017.244 ↗
45. Lee, H., Kang, M., Song, J. & Hwang, K. Pix2Pix-based data augmentation method for building an image dataset of black ice. SSRN https://doi.org/10.2139/ssrn.4044502 (2022).10.2139/ssrn.4044502 ↗
46. Fawakherji, M., Suriani, V., Nardi, D. & Bloisi, D. D. Shape and style GAN-based multispectral data augmentation for crop/weed segmentation in precision farming. Crop Prot. 184, 106848 (2024).10.1016/j.cropro.2024.106848 ↗
47. Yan, R., Peng, L., Xiao, S. & Yao, G. Primitive representation learning for scene text recognition. In Proceedings of theIEEE/CVF conference on computer vision and pattern recognition (pp. 284−293) (2021).10.1109/CVPR46437.2021.00035 ↗
48. Borisyuk, F., Gordo, A. & Sivakumar, V. Rosetta: Large scale system for text detection and recognition in images. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 71−79 (ACM, 2018).10.1145/3219819.3219861 ↗
49. Fang, S., Xie, H., Wang, Y., Mao, Z. & Zhang, Y. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7098−7107 (IEEE, 2021).10.1109/CVPR46437.2021.00702 ↗
50. Wan, Z., He, M., Chen, H., Bai, X. & Yao, C. Textscanner: Reading characters in order for robust scene text recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, 12120−12127 (IEEE, 2020).10.1609/aaai.v34i07.6891 ↗
51. Wang, Z. et al. Symmetrical linguistic feature distillation with clip for scene text recognition. In Proceedings of the 31st ACM International Conference on Multimedia, 509−518 (ACM, 2023).10.1145/3581783.3611769 ↗
52. Aberdam, A. et al. Clipter: Looking at the bigger picture in scene text recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision,21706−21717 (IEEE, 2023).10.1109/ICCV51070.2023.01984 ↗
53. Cui, M., Wang, W., Zhang, J. & Wang, L. Representation and correlation enhanced encoder-decoder framework for scene text recognition. In Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part IV 16, 156−170 (Springer International Publishing, 2021).10.1007/978-3-030-86337-1_11 ↗
54. Zhao, S., Quan, R., Zhu, L. & Yang, Y. CLIP4STR: a simple baseline for scene text recognition with pre-trained vision-language model. In IEEE Transactions on Image Processing, 6893−6904 (IEEE, 2024).10.1109/TIP.2024.3512354 ↗
55. Liu, Y. et al. Multilingual denoising pre-training for neural machine translation. Trans. Assoc. Comput. Linguist. 8, 726–742 (2020).10.1162/tacl_a_00343 ↗
56. Xu, H., Liu, Q., van Genabith, J., Xiong, D. & Zhang, M. Multi-head highly parallelized LSTM decoder for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 273−282 (2021).10.18653/v1/2021.acl-long.23 ↗
57. Ramadhan, T. I., Ramadhan, N. G. & Supriatman, A. Implementation of neural machine translation for english-sundanese language using long short term memory (lstm). Build. Inform. Technol. Sci. (BITS 4, 1438–1446 (2022).
58. Velickovic, P. et al. Graph attention networks. stat 1050, 10–48550 (2017).
59. Baek, J. et al. What is wrong with scene text recognition model comparisons? dataset and model analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4715−4723 (IEEE, 2019).10.1109/ICCV.2019.00481 ↗
60. Xu, Y. et al. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1192−1200 (ACM, 2020).10.1145/3394486.3403172 ↗
61. Cheema, M. D. A., Shaiq, M. D., Mirza, F., Kamal, A. & Naeem, M. A. Adapting multilingual vision language transformers for low-resource Urdu optical character recognition (OCR). PeerJ Comput. Sci. 10, e1964 (2024).10.7717/peerj-cs.1964 ↗
62. Manoranjitham, T. Phonetic-aware word reconstruction in ancient tamil inscriptions using adaptive graph networks. Knowl. Based Syst. 327, 114087 (2025).10.1016/j.knosys.2025.114087 ↗
63. Agasar, B. & Mukarambi, G. Hybrid method for enhancement of low-resolution ancient kannada stone inscription images. Digit. Appl. Archaeol. Cult. Herit. 35, e00385 (2024).
64. Wang, Z., Li, Y. & Li, H. Chinese inscription restoration based on artificial intelligent models. npj Herit. Sci. 13, 326 (2025).10.1038/s40494-025-01900-x ↗
65. Kim, G. et al. Ocr-free document understanding transformer. In Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes In Computer Science, (eds. Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T.) 13688 (Cham: Springer Nature Switzerland, 2022).
66. Koç, C. & Özyurt, F. An examination of synthetic images produced with DCGAN according to the size of data and epoch. Firat Univ. J. Exp. Comput. Eng. 2, 32–37 (2023).10.5505/fujece.2023.69885 ↗