Scene Text Detection Based on Multi-Headed Self-Attention Using Shifted Windows
Source abstract
Abstract
Scene text detection has become a popular topic in computer vision research. Most of the current research is based on deep learning, using Convolutional Neural Networks (CNNs) to extract the visual features of images. However, due to the limitations of convolution kernel size, CNNs can only extract local features of images with small perceptual fields, and they cannot obtain more global features. In this paper, to improve the accuracy of scene text detection, a feature enhancement module is added to the text detection model. This module acquires global features of an image by computing the multi-headed self-attention of the feature map. The improved model extracts local features using CNNs, while extracting global features through the feature enhancement module. The features extracted by both of these are then fused to ensure that visual features at different levels of the image are extracted. A shifted window is used in the calculation of the self-attention, which reduces the computational complexity from the second power of the input image width-height product to the first power. Experiments are conducted on the multi-oriented text dataset ICDAR2015 and the multi-language text dataset MSRA-TD500. Compared with the pre-improvement method DBNet, the F1-score improves by 0.5% and 3.5% on ICDAR2015 and MSRA-TD500, respectively, indicating the effectiveness of the model improvement.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques · Vehicle License Plate Recognition · Image Processing and 3D Reconstruction
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer and Electronic Information, Guangxi University, Nanning 530004, China
Funding
National Natural Science Foundation of China
References (39)
Show source-provided references
1. Cao, D., Zhong, Y., Wang, L., He, Y., and Dang, J. (2020). Scene Text Detection in Natural Images: A Review. Symmetry, 12.10.3390/sym12121956 ↗
2. Ibravim, M., Li, Y., and Hamdulla, A. (2022). Scene Text Detection Based on Two-Branch Feature Extraction. Sensors, 22.10.3390/s22166262 ↗
3. Hassan, E., and Lekshmi, V. (2022). Scene Text Detection Using Attention with Depthwise Separable Convolutions. Appl. Sci., 12.10.3390/app12136425 ↗
4. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
5. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C., and Berg, A.C. (2016, January 11–14). SSD: Single shot multibox detector. Proceedings of the Computer Vision—ECCV 2016, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
6. R-FCN: Object detection via region-based fully convolutional networks
7. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
8. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
9. Lin, T., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
10. Li, Y., Qi, H., Dai, J., Ji, X., and Wei, Y. (2017, January 21–26). Fully convolutional instance-aware semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.472 ↗
11. Attention is all you need
12. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
13. Liao, M., Shi, B., Bai, X., Wang, X., and Liu, W. (2017, January 4–9). Textboxes: A fast text detector with a single deep neural network. Proceedings of the Thirty-first AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11196 ↗
14. Textboxes++: A single-shot oriented scene text detector10.1109/TIP.2018.2825107 ↗
15. Wang, Y., Xie, H., Zha, Z., Xing, M., Fu, Z., and Zhang, Y. (2020, January 13–19). ContourNet: Taking a further step toward accurate arbitrary-shaped scene text detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01177 ↗
16. Xiao, L., Zhou, P., Xu, K., and Zhao, X. (2021). Multi-Directional Scene Text Detection Based on Improved YOLOv3. Sensors, 21.10.3390/s21144870 ↗
17. He, M., Liao, M., Yang, Z., Zhong, H., Tang, J., Cheng, W., Yao, C., Wang, Y., and Bai, X. (2021, January 20–25). MOST: A multi-oriented scene text detector with localization refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00870 ↗
18. Wang, W., Xie, E., Li, X., Hou, W., Lu, T., Yu, G., and Shao, S. (2019, January 15–20). Shape robust text detection with progressive scale expansion network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00956 ↗
19. Liao, M., Wan, Z., Yao, C., Chen, K., and Bai, X. (2020, January 7–12). Real-time scene text detection with differentiable banalization. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6812 ↗
20. Zhu, Y., Chen, J., Liang, L., Kuang, Z., Jin, L., and Zhang, W. (2021, January 20–25). Fourier contour embedding for arbitrary-shaped text detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00314 ↗
21. Cao, D., Dang, J., and Zhong, Y. (2021). Towards Accurate Scene Text Detection with Bidirectional Feature Pyramid Network. Symmetry, 13.10.3390/sym13030486 ↗
22. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
23. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
24. Gupta, A., Vedaldi, A., and Zisserman, A. (2016, January 27–30). Synthetic data for text localization in natural images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.254 ↗
25. Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V.R., and Lu, S. (2015, January 23–26). ICDAR 2015 competition on robust reading. Proceedings of the 2015 13th International Conference on Document Analysis and Recognition (ICDAR), Tunis, Tunisia.10.1109/ICDAR.2015.7333942 ↗
26. Yao, C., Bai, X., Liu, W., Ma, Y., and Tu, Z. (2012, January 16–21). Detecting texts of arbitrary orientations in natural images. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.
27. A Unified Framework for Multioriented Text Detection and Recognition10.1109/TIP.2014.2353813 ↗
28. Tian, Z., Huang, W., He, T., He, P., and Qiao, Y. (2016, January 11–14). Detecting Text in Natural Image with Connectionist Text Proposal Network. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_4 ↗
29. He, P., Huang, W., He, T., Zhu, Q., Qiao, Y., and Li, X. (2017, January 22–29). Single shot text detector with regional attention. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.331 ↗
30. Hu, H., Zhang, C., Luo, Y., Wang, Y., Han, J., and Ding, E. (2017, January 22–29). WordSup: Exploiting word annotations for character based text detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.529 ↗
31. Lyu, P., Yao, C., Wu, W., Yan, S., and Bai, X. (2018, January 18–23). Multi-oriented scene text detection via corner localization and region segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00788 ↗
32. Liao, M., Zhu, Z., Shi, B., Xia, G., and Bai, X. (2018, January 18–23). Rotation-sensitive regression for oriented scene text detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00619 ↗
33. Liu, Z., Lin, G., Yang, S., Feng, J., Lin, W., and Goh, W.L. (2018, January 18–23). Learning markov clustering networks for scene text detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00725 ↗
34. Long, S., Ruan, J., Zhang, W., He, X., Wu, W., and Yao, C. (2018, January 8–14). Textsnake: A flexible representation for detecting text of arbitrary shapes. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01216-8_2 ↗
35. Tian, Z., Shu, M., Lyu, P., Li, R., Zhou, C., Shen, X., and Jia, J. (2019, January 15–20). Learning shape-aware embedding for scene text detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00436 ↗
36. Pan++: Towards efficient and accurate end-to-end spotting of arbitrarily-shaped text
37. Chen, Z., Wang, J., Wang, W., Chen, G., Xie, E., Luo, P., and Lu, T. (2023). FAST: Faster arbitrarily-shaped text detector with minimalist kernel representation. arXiv.
38. He, W., Zhang, X., Yin, F., and Liu, C. (2017, January 22–29). Deep direct regression for multi-oriented scene text detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.87 ↗
39. Baek, Y., Lee, B., Han, D., Yun, S., and Lee, H. (2019, January 15–20). Character region awareness for text detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00959 ↗