From Detection to Understanding: A Systematic Survey of Deep Learning for Scene Text Processing
Source abstract
Abstract
Scene text understanding, serving as a cornerstone technology for autonomous navigation, document digitization, and accessibility tools, has witnessed a paradigm shift from traditional methods relying on handcrafted features and multi-stage processing pipelines to contemporary deep learning frameworks capable of learning hierarchical representations directly from raw image inputs. This survey distinctly categorizes modern scene text recognition (STR) methodologies into three principal paradigms: two-stage detection frameworks that employ region proposal networks for precise text localization, single-stage detectors designed to optimize computational efficiency, and specialized architectures tailored to handle arbitrarily shaped text through geometric-aware modeling techniques. Concurrently, an in-depth analysis of text recognition paradigms elucidates the evolutionary trajectory from connectionist temporal classification (CTC) and sequence-to-sequence models to transformer-based architectures, which excel in contextual modeling and demonstrate superior performance. In contrast to prior surveys, this work uniquely emphasizes several key differences and contributions. Firstly, it provides a comprehensive and systematic taxonomy of STR methods, explicitly highlighting the trade-offs between detection accuracy, computational efficiency, and geometric adaptability across different paradigms. Secondly, it delves into the nuances of text recognition, illustrating how transformer-based models have revolutionized the field by capturing long-range dependencies and contextual information, thereby addressing challenges in recognizing complex text layouts and multilingual scripts. Furthermore, the survey pioneers the exploration of critical research frontiers, such as multilingual text adaptation, enhancing model robustness against environmental variations (e.g., lighting conditions, occlusions), and devising data-efficient learning strategies to mitigate the dependency on large-scale annotated datasets. By synthesizing insights from technical advancements across 28 benchmark datasets and standardized evaluation protocols, this study offers researchers a holistic perspective on the current state-of-the-art, persistent challenges, and promising avenues for future research, with the ultimate goal of achieving human-level scene text comprehension.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Xinjiang Normal University, Urumqi 830054, China
Library of Xinjiang Normal University, Urumqi 830017, China
Funding
National Natural Science Foundation of China · Doctoral Research Foundation of Xinjiang Normal University · Xinjiang Normal University Youth Top Talents Project · Science and Technology Program Project (General Program) of Xinjiang Uygur Autonomous Region · Xinjiang Key Research and Development Program
References (100+)
Show source-provided references
1. Liu, Y., Chen, H., Shen, C., He, T., Jin, L., and Wang, L. (2020, January 13–19). ABCNet: Real-time scene text spotting with adaptive bezier-curve network. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00983 ↗
2. Reading text in the wild with convolutional neural networks10.1007/s11263-015-0823-z ↗
3. Kittenplon, Y., Lavi, I., Fogel, S., Bar, Y., Manmatha, R., and Perona, P. (2022, January 19–24). Towards weakly-supervised text spotting using a multi-task transformer. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00456 ↗
4. Long, S., Ruan, J., Zhang, W., He, X., Wu, W., and Yao, C. (2018). Textsnake: A flexible representation for detecting text of arbitrary shapes. arXiv.10.1007/978-3-030-01216-8_2 ↗
5. Mask TextSpotter v3: Segmentation proposal network for robust scene text spotting
6. TextBlockV2: Towards precise-detection-free scene text spotting with pre-trained language model10.1145/3734872 ↗
7. Xu, H., Wang, Q.-F., Li, Z., Shi, Y., and Zhou, X.-D. (2025). TextTriangle: An end-to-end textspotter with piecewise linear alignment. Int. J. Doc. Anal. Recognit.10.1007/s10032-025-00517-x ↗
8. Raisi, Z., Naiel, M.A., Fieguth, P., Wardell, S., and Zelek, J. (2020). Text detection and recognition in the wild: A review. arXiv.
9. Faster R-CNN: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
10. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask R-CNN. arXiv.10.1109/ICCV.2017.322 ↗
11. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016). You only look once: Unified, real-time object detection. arXiv.10.1109/CVPR.2016.91 ↗
12. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A.C. (2016). SSD: Single shot multibox detector. Lecture Notes in Computer Science, Proceedings of the Computer Vision–ECCV 2016, Amsterdam, The Netherlands, 8–16 October 2016, Springer.10.1007/978-3-319-46448-0_2 ↗
13. Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V.R., and Lu, S. (2015, January 23–26). ICDAR 2015 competition on robust reading. Proceedings of the 2015 13th International Conference on Document Analysis and Recognition (ICDAR), Tunis, Tunisia.10.1109/ICDAR.2015.7333942 ↗
14. Ch’ng, C.K., and Chan, C.S. (2017, January 9–15). Total-text: A comprehensive dataset for scene text detection and recognition. Proceedings of the 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan.10.1109/ICDAR.2017.157 ↗
15. Mishra, A., Alahari, K., and Jawahar, C.V. (2012, January 3–7). Scene text recognition using higher order language priors. Proceedings of the BMVC-British Machine Vision Conference, Surrey, UK.
16. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning transferable visual models from natural language supervision. arXiv.
17. Fully convolutional networks for semantic segmentation10.1109/TPAMI.2016.2572683 ↗
18. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143891 ↗
19. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
20. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition10.1109/TPAMI.2016.2646371 ↗
21. Liu, X., Liang, D., Yan, S., Chen, D., Qiao, Y., and Yan, J. (2018, January 18–23). FOTS: Fast oriented text spotting with a unified network. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00595 ↗
22. Mask TextSpotter: An end-to-end trainable neural network for spotting text with arbitrary shapes10.1109/TPAMI.2019.2937086 ↗
23. SPTS v2: Single-point text spotting10.1109/TPAMI.2023.3312285 ↗
24. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. arXiv.
25. TrOCR: Transformer-based optical character recognition with pre-trained models
26. Du, Y., Chen, Z., Jia, C., Yin, X., Zheng, T., Li, C., Du, Y., and Jiang, Y.-G. (2022). Svtr: Scene text recognition with a single visual model. arXiv.10.24963/ijcai.2022/124 ↗
27. Epshtein, B., Ofek, E., and Wexler, Y. (2010, January 13–18). Detecting text in natural scenes with stroke width transform. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPR.2010.5540041 ↗
28. Robust wide-baseline stereo from maximally stable extremal regions10.1016/j.imavis.2004.02.006 ↗
29. Wang, K., Babenko, B., and Belongie, S. (2011, January 6–13). End-to-end scene text recognition. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.
30. Arbitrary-oriented scene text detection via rotation proposals10.1109/TMM.2018.2818020 ↗
31. Detecting text in natural image with connectionist text proposal network
32. Zhou, X., Yao, C., Wen, H., Wang, Y., Zhou, S., He, W., and Liang, J. (2017, January 21–26). EAST: An efficient and accurate scene text detector. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.283 ↗
33. Zhang, S.-X., Zhu, X., Yang, C., Wang, H., and Yin, X.-C. (2021, January 11–17). Adaptive boundary proposal network for arbitrary shape text detection. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00134 ↗
34. Arbitrary shape text detection via boundary transformer10.1109/TMM.2023.3286657 ↗
35. Dptext-detr: Towards better scene text detection with dynamic points in transformer
36. Zheng, J., Zhang, L., Wu, Y., and Zhao, C. (2024, January 14–19). BPDO: Boundary points dynamic optimization for arbitrary shape scene text detection. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10447371 ↗
37. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2020). Deformable detr: Deformable transformers for end-to-end object detection. arXiv.
38. TextBoxes: A fast text detector with a single deep neural network
39. Textboxes++: A single-shot oriented scene text detector10.1109/TIP.2018.2825107 ↗
40. AB-LSTM: Attention-based bidirectional LSTM model for scene text detection
41. Wang, W., Xie, E., Song, X., Zang, Y., Wang, W., Lu, T., Yu, G., and Shen, C. (November, January 27). Efficient and accurate arbitrary-shaped text detection with pixel aggregation network. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00853 ↗
42. Wang, W., Xie, E., Li, X., Hou, W., Lu, T., Yu, G., and Shao, S. (2019). Shape robust text detection with progressive scale expansion network. arXiv.10.1109/CVPR.2019.00956 ↗
43. MFECN: Multi-level feature enhanced cumulative network for scene text detection
44. Real-time scene text detection with differentiable binarization
45. Milletari, F., Navab, N., and Ahmadi, S.-A. (2016, January 25–28). V-Net: Fully convolutional neural networks for volumetric medical image segmentation. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.10.1109/3DV.2016.79 ↗
46. Focal loss for dense object detection10.1109/TPAMI.2018.2858826 ↗
47. Kingma, D.P. (2014). Adam: A method for stochastic optimization. arXiv.
48. A large chinese text dataset in the wild10.1007/s11390-019-1923-y ↗
49. Boundary-Preserving Mask R-CNN
50. TextField: Learning a deep direction field for irregular scene text detection10.1109/TIP.2019.2900589 ↗
51. Boundary TextSpotter: Toward arbitrary-shaped scene text spotting10.1109/TIP.2022.3206615 ↗
52. Shi, B., Wang, X., Lyu, P., Yao, C., and Bai, X. (2016, January 27–30). Robust scene text recognition with automatic rectification. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.452 ↗
53. ASTER: An attentional scene text recognizer with flexible rectification10.1109/TPAMI.2018.2848939 ↗
54. Show, attend and read: A simple and strong baseline for irregular text recognition
55. Bai, F., Cheng, Z., Niu, Y., Pu, S., and Zhou, S. (2018, January 18–23). Edit probability for scene text recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00163 ↗
56. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009, January 14–18). Curriculum learning. Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada.10.1145/1553374.1553380 ↗
57. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
58. Vision transformer for fast and efficient scene text recognition
59. Huang, M., Liu, Y., Peng, Z., Liu, C., Lin, D., Zhu, S., Yuan, N., Ding, K., and Jin, L. (2022, January 18–24). SwinTextSpotter: Scene text spotting via better synergy between text detection and text recognition. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00455 ↗
60. Image as a language: Revisiting scene text recognition via balanced, unified and synchronized vision-language reasoning network
61. Fang, S., Xie, H., Wang, Y., Mao, Z., and Zhang, Y. (2021, January 20–25). Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00702 ↗
62. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
63. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
64. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
65. Learning long-term dependencies with gradient descent is difficult10.1109/72.279181 ↗
66. Wang, S., Li, B.Z., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv.
67. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
68. Han, S., Pool, J., Tran, J., and Dally, W. (2015). Learning both weights and connections for efficient neural network. arXiv.
69. Sheng, F., Chen, Z., and Xu, B. (2019, January 20–25). NRTR: A no-recurrence sequence-to-sequence model for scene text recognition. Proceedings of the 2019 International Conference on Document Analysis and Recognition (ICDAR), Sydney, Australia.10.1109/ICDAR.2019.00130 ↗
70. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.10.3115/v1/D14-1179 ↗
71. Li, H., Wang, P., and Shen, C. (2017). Towards end-to-end text spotting with convolutional recurrent neural networks. arXiv.10.1109/ICCV.2017.560 ↗
72. E2E-MLT—An unconstrained end-to-end method for multi-language scene text
73. Char-net: A character-aware neural network for distorted scene text recognition
74. End-to-end object detection with transformers
75. Girshick, R. (2015, January 7–13). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.169 ↗
76. Feng, W., He, W., Yin, F., Zhang, X.-Y., and Liu, C.-L. (November, January 27). Textdragon: An end-to-end framework for arbitrary shaped text spotting. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00917 ↗
77. MANGO: A mask attention guided one-stage scene text spotter
78. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
79. Exploring the capabilities and limitations of GPT and Chat GPT in natural language processing
80. Yu, W., Liu, Y., Hua, W., Jiang, D., Ren, B., and Bai, X. (2023, January 18–22). Turning a clip model into a scene text detector. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00674 ↗
81. Song, S., Wan, J., Yang, Z., Tang, J., Cheng, W., Bai, X., and Yao, C. (2022, January 19–24). Vision-language pre-training for boosting scene text detectors. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01523 ↗
82. Nayef, N., Patel, Y., Busta, M., Chowdhury, P.N., Karatzas, D., Khlif, W., Matas, J., Pal, U., Burie, J.-C., and Liu, C.-L. (2019, January 20–25). ICDAR2019 robust reading challenge on multi-lingual scene text detection and recognition—RRC-MLT-2019. Proceedings of the 2019 International Conference on Document Analysis and Recognition (ICDAR), Sydney, Australia.10.1109/ICDAR.2019.00254 ↗
83. Baek, J., Matsui, Y., and Aizawa, K. (2021, January 20–25). What if we only use real datasets for scene text recognition? toward scene text recognition with fewer labels. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00313 ↗
84. Yuliang, L., Lianwen, J., Shuaitao, Z., and Sheng, Z. (2017). Detecting curve text in the wild: New dataset and new solution. arXiv.
85. Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., and Hassner, T. (2021, January 20–25). TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00869 ↗
86. Long, S., Qin, S., Panteleev, D., Bissacco, A., Fujii, Y., and Raptis, M. (2022). Towards end-to-end unified scene text detection and layout analysis. arXiv.10.1109/CVPR52688.2022.00112 ↗
87. Baek, J., Kim, G., Lee, J., Park, S., Han, D., Yun, S., Oh, S.J., and Lee, H. (November, January 27). What is wrong with scene text recognition model comparisons? dataset and model analysis. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00481 ↗
88. Scene text detection and recognition: The deep learning era10.1007/s11263-020-01369-0 ↗
89. Wang, W., Xie, E., Sun, P., Wang, W., Tian, L., Shen, C., and Luo, P. (2019). Textsr: Content-aware text super-resolution guided by recognition. arXiv.
90. Transforming Scene Text Detection and Recognition: A Multi-Scale End-to-End Approach With Transformer Framework10.1109/ACCESS.2024.3375497 ↗
91. Cheng, C., Wang, P., Da, C., Zheng, Q., and Yao, C. (2023, January 2–6). LISTER: Neighbor decoding for length-insensitive scene text recognition. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01790 ↗
92. Lyu, P., Zhang, C., Liu, S., Qiao, M., Xu, Y., Wu, L., Yao, K., Han, J., Ding, E., and Wang, J. (2022). Maskocr: Text recognition with masked encoder-decoder pretraining. arXiv.
93. Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv.
94. Zheng, C., Li, H., Rhee, S.-M., Han, S., Han, J.-J., and Wang, P. (2022, January 18–24). Pushing the performance limit of scene text recognizer without human annotation. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01372 ↗
95. Finn, C., Abbeel, P., and Levine, S. (2017, January 6–11). Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
96. Bhunia, A.K., Ghose, S., Kumar, A., Chowdhury, P.N., Sain, A., and Song, Y.-Z. (2021, January 20–25). MetaHTR: Towards writer-adaptive handwritten text recognition. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01557 ↗
97. Jaderberg, M., Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Synthetic data and artificial neural networks for natural scene text recognition. arXiv.
98. Nguyen, N., Nguyen, T., Tran, V., Tran, M.-T., Ngo, T.D., Nguyen, T.H., and Hoai, M. (2021, January 20–25). Dictionary-guided scene text recognition. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00730 ↗
99. Turning a CLIP Model into a Scene Text Spotter10.1109/TPAMI.2024.3379828 ↗
100. Duan, C., Fu, P., Guo, S., Jiang, Q., and Wei, X. (2024, January 16–20). Odm: A text-image further alignment pre-training approach for scene text detection and spotting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01476 ↗