Scene Text Detection Using Attention with Depthwise Separable Convolutions
Source abstract
Abstract
In spite of significant research efforts, the existing scene text detection methods fall short of the challenges and requirements posed in real-life applications. In natural scenes, text segments exhibit a wide range of shape complexities, scale, and font property variations, and they appear mostly incidental. Furthermore, the computational requirement of the detector is an important factor for real-time operation. To address the aforementioned issues, the paper presents a novel scene text detector using a deep convolutional network which efficiently detects arbitrary oriented and complex-shaped text segments from natural scenes and predicts quadrilateral bounding boxes around text segments. The proposed network is designed in a U-shape architecture with the careful incorporation of skip connections to capture complex text attributes at multiple scales. For addressing the computational requirement of the input processing, the proposed scene text detector uses the MobileNet model as the backbone that is designed on depthwise separable convolutions. The network design is integrated with text attention blocks to enhance the learning ability of our detector, where the attention blocks are based on efficient channel attention. The network is trained in a multi-objective formulation supported by a novel text-aware non-maximal procedure to generate final text bounding box predictions. On extensive evaluations on ICDAR2013, ICDAR2015, MSRA-TD500, and COCOText datasets, the paper reports detection F-scores of 0.910, 0.879, 0.830, and 0.617, respectively.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques · Vehicle License Plate Recognition · Image Processing and 3D Reconstruction
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Kuwait College of Science and Technology, Doha 35004, Kuwait
Funding
Kuwait Foundation for the Advancement of Sciences
References (75)
Show source-provided references
1. Text extraction from natural scene image: A survey10.1016/j.neucom.2013.05.037 ↗
2. Epshtein, B., Ofek, E., and Wexler, Y. (2010, January 13–18). Detecting text in natural scenes with stroke width transform. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPR.2010.5540041 ↗
3. Neumann, L., and Matas, J. (2010, January 8–12). A Method for Text Localization and Recognition in Real-World Images. Proceedings of the ACCV, Queenstown, New Zealand.
4. Lienhart, R., and Stuber, F. (February, January 28). Automatic text recognition in digital videos. Proceedings of the Electronic Imaging, San Jose, CA, USA.
5. Cai, M., Song, J., and Lyu, M.R. (2002, January 22–25). A new approach for video text detection. Proceedings of the International Conference on Image Processing, Rochester, NY, USA.
6. Agnihotri, L., and Dimitrova, N. (1999, January 22). Text detection for video analysis. Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries (CBAIVL’99), Fort Collins, CO, USA.
7. Ezaki, N., Bulacu, M., and Schomaker, L. (2004, January 26). Text detection from natural scene images: Towards a system for visually impaired persons. Proceedings of the of the 17th International Conference on Pattern Recognition (ICPR 2004), Cambridge, UK.10.1109/ICPR.2004.1334351 ↗
8. Neumann, L., and Matas, J.E.S. (2012, January 16–21). Real-time scene text localization and recognition. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248097 ↗
9. Robust Text Detection in Natural Scene Images10.1109/TPAMI.2013.182 ↗
10. Cho, H., Sung, M., and Jun, B. (2016, January 27–30). Canny Text Detector: Fast and Robust Scene Text Localization Algorithm. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.388 ↗
11. Li, Y., and Lu, H. (2012, January 11–15). Scene text detection via stroke width. Proceedings of the of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan.
12. Gomez, L., and Karatzas, D. (2014, January 24–28). MSER-based real-time text detection and tracking. Proceedings of the 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden.10.1109/ICPR.2014.536 ↗
13. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
14. Scene text detection and recognition with advances in deep learning: A survey10.1007/s10032-019-00320-5 ↗
15. Scene text detection and recognition: The deep learning era10.1007/s11263-020-01369-0 ↗
16. (2022, May 08). Available online: https://rrc.cvc.uab.es/.
17. (2022, May 08). Available online: https://paperswithcode.com/sota/scene-text-detection-on-msra-td500.
18. (2022, May 08). Available online: https://paperswithcode.com/sota/scene-text-detection-on-coco-text.
19. Howard, A., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
20. Ba, J., Mnih, V., and Kavukcuoglu, K. (2014). Multiple object recognition with visual attention. arXiv.
21. Jaderberg, M., Simonyan, K., Zisserman, A., and Kavukcuoglu, K. (2015, January 7–12). Spatial Transformer Networks. Proceedings of the of the 28th International Conference on Neural Information Processing Systems (NIPS’15), Montreal, QC, Canada.
22. Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8–14). Cbam: Convolutional block attention module. Proceedings of the of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
23. Park, J., Woo, S., Lee, J.Y., and Kweon, I.S. (2018). Bam: Bottleneck attention module. arXiv.
24. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., and Hu, Q. (2019). ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks. arXiv.10.1109/CVPR42600.2020.01155 ↗
25. Delakis, M., and Garcia, C. (2008, January 22–25). Text Detection with Convolutional Neural Networks. Proceedings of the VISAPP (2), Madeira, Portugal.
26. Wang, T., Wu, D.J., Coates, A., and Ng, A.Y. (2012, January 11–15). End-to-end text recognition with convolutional neural networks. Proceedings of the of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan.
27. Jaderberg, M., Vedaldi, A., and Zisserman, A. (2014, January 6–12). Deep Features for Text Spotting. Proceedings of the ECCV, Zurich, Switzerland.10.1007/978-3-319-10593-2_34 ↗
28. Girshick, R.B. (2015, January 7–13). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.169 ↗
29. Dai, J., Li, Y., He, K., and Sun, J. (2016, January 5–10). R-FCN: Object Detection via Region-based Fully Convolutional Networks. Proceedings of the Advances in Neural Information Processing Systems 29 (NIPS 2016), Barcelona, Spain.
30. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S.E., Fu, C., and Berg, A.C. (2015). SSD: Single Shot MultiBox Detector. arXiv.10.1007/978-3-319-46448-0_2 ↗
31. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask r-cnn. Proceedings of the of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
32. Zhou, X., Yao, C., Wen, H., Wang, Y., Zhou, S., He, W., and Liang, J. (2017, January 21–26). EAST: An Efficient and Accurate Scene Text Detector. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.283 ↗
33. Zhang, Z., Zhang, C., Shen, W., Yao, C., Liu, W., and Bai, X. (2016, January 27–30). Multi-oriented Text Detection with Fully Convolutional Networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.451 ↗
34. Lyu, P., Liao, M., Yao, C., Wu, W., and Bai, X. (2018, January 8–14). Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes. Proceedings of the of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_5 ↗
35. Text-Attentional Convolutional Neural Network for Scene Text Detection10.1109/TIP.2016.2547588 ↗
36. TextBoxes++: A Single-Shot Oriented Scene Text Detector10.1109/TIP.2018.2825107 ↗
37. Qin, X., Zhou, Y., Guo, Y., Wu, D., Tian, Z., Jiang, N., Wang, H., and Wang, W. (2021, January 20–24). Mask is All You Need: Rethinking Mask R-CNN for Dense and Arbitrary-Shaped Scene Text Detection. Proceedings of the 29th ACM International Conference on Multimedia, Virtual Event.10.1145/3474085.3475178 ↗
38. Liao, M., Shi, B., Bai, X., Wang, X., and Liu, W. (2016). TextBoxes: A Fast Text eastDetector with a Single Deep Neural Network. arXiv.10.1609/aaai.v31i1.11196 ↗
39. Zhu, X., Jiang, Y., Yang, S., Wang, X., Li, W., Fu, P., Wang, H., and Luo, Z. (2017, January 9–15). Deep Residual Text Detection Network for Scene Text. Proceedings of the 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan.10.1109/ICDAR.2017.137 ↗
40. Tian, Z., Huang, W., He, T., He, P., and Qiao, Y. (2016). Detecting Text in Natural Image with Connectionist Text Proposal Network. arXiv.10.1007/978-3-319-46484-8_4 ↗
41. Zhong, Z., Jin, L., Zhang, S., and Feng, Z. (2016). DeepText: A Unified Framework for Text Proposal Generation and Text Detection in Natural Images. arXiv.10.1109/ICASSP.2017.7952348 ↗
42. Yao, C., Bai, X., Shi, B., and Liu, W. (2014, January 23–28). Strokelets: A learned multi-scale representation for scene text recognition. Proceedings of the of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.515 ↗
43. Wang, X., Jiang, Y., Luo, Z., Liu, C., Choi, H., and Kim, S. (2019, January 15–20). Arbitrary Shape Scene Text Detection With Adaptive Text Region Representation. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00661 ↗
44. Xing, L., Tian, Z., Huang, W., and Scott, M.R. (November, January 27). Convolutional character networks. Proceedings of the of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
45. PRPN: Progressive region prediction network for natural scene text detection10.1016/j.knosys.2021.107767 ↗
46. Deng, D., Liu, H., Li, X., and Cai, D. (2018, January 2–7). Pixellink: Detecting scene text via instance segmentation. Proceedings of the of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12269 ↗
47. Xie, E., Zang, Y., Shao, S., Yu, G., Yao, C., and Li, G. (February, January 27). Scene text detection with supervised pyramid context network. Proceedings of the of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
48. Huang, Z., Zhong, Z., Sun, L., and Huo, Q. (2019, January 7–11). Mask R-CNN with pyramid attention network for scene text detection. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV.2019.00086 ↗
49. He, W., Zhang, X.Y., Yin, F., and Liu, C.L. (2017). Deep Direct Regression for Multi-Oriented Scene Text Detection. arXiv.10.1109/ICCV.2017.87 ↗
50. Shi, B., Bai, X., and Belongie, S. (2017, January 21–26). Detecting Oriented Text in Natural Images by Linking Segments. Proceedings of the of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.371 ↗
51. Lyu, P., Yao, C., Wu, W., Yan, S., and Bai, X. (2018, January 18–23). Multi-oriented scene text detection via corner localization and region segmentation. Proceedings of the of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00788 ↗
52. Detecting multi-oriented text with corner-based region proposals10.1016/j.neucom.2019.01.013 ↗
53. Zhang, C., Liang, B., Huang, Z., En, M., Han, J., Ding, E., and Ding, X. (2019, January 15–20). Look More Than Once: An Accurate Detector for Text of Arbitrary Shapes. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01080 ↗
54. Hu, H., Zhang, C., Luo, Y., Wang, Y., Han, J., and Ding, E. (2017, January 22–29). WordSup: Exploiting Word Annotations for Character Based Text Detection. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.529 ↗
55. Fragoso, V., Gauglitz, S., Zamora, S., Kleban, J., and Turk, M. (2011, January 5–7). TranslatAR: A mobile augmented reality translator. Proceedings of the 2011 IEEE Workshop on Applications of Computer Vision (WACV), Kona, HI, USA.10.1109/WACV.2011.5711545 ↗
56. Petter, M., Fragoso, V., Turk, M., and Baur, C. (2011, January 6–13). Automatic text detection for mobile augmented reality translation. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.10.1109/ICCVW.2011.6130221 ↗
57. Scene text recognition in mobile applications by character descriptor and structure configuration10.1109/TIP.2014.2317980 ↗
58. Fractals based multi-oriented text detection system for recognition in mobile video images10.1016/j.patcog.2017.03.018 ↗
59. Fu, K., Sun, L., Kang, X., and Ren, F. (2019, January 4–7). Text Detection for Natural Scene based on MobileNet V2 and U-Net. Proceedings of the 2019 IEEE International Conference on Mechatronics and Automation (ICMA), Tianjin, China.10.1109/ICMA.2019.8816384 ↗
60. Shrivastava, A., Gupta, A., and Girshick, R. (2016, January 27–30). Training region-based object detectors with online hard example mining. Proceedings of the of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.89 ↗
61. Focal Loss for Dense Object Detection10.1109/TPAMI.2018.2858826 ↗
62. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-Excitation Networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
63. Neubeck, A., and Van Gool, L. (2006, January 20–24). Efficient non-maximum suppression. Proceedings of the 18th International Conference on Pattern Recognition (ICPR’06), Hong Kong, China.10.1109/ICPR.2006.479 ↗
64. Bodla, N., Singh, B., Chellappa, R., and Davis, L.S. (2017, January 22–29). Soft-NMS–improving object detection with one line of code. Proceedings of the of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.593 ↗
65. Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., i Bigorda, L.G., Mestre, S.R., Mas, J., Mota, D.F., Almazan, J.A., and De Las Heras, L.P. (2013, January 25–28). ICDAR 2013 robust reading competition. Proceedings of the 2013 12th International Conference on Document Analysis and Recognition, Washington, DC, USA.10.1109/ICDAR.2013.221 ↗
66. Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V.R., and Lu, S. (2015, January 23–26). ICDAR 2015 competition on robust reading. Proceedings of the 2015 13th International Conference on Document Analysis and Recognition (ICDAR), Tunis, Tunisia.10.1109/ICDAR.2015.7333942 ↗
67. Veit, A., Matera, T., Neumann, L., Matas, J., and Belongie, S. (2016). Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv.
68. Lin, T.Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., and Dollár, P. (2014). Microsoft COCO: Common Objects in Context. arXiv.10.1007/978-3-319-10602-1_48 ↗
69. Yao, C., Bai, X., Liu, W., Ma, Y., and Tu, Z. (2012, January 16–21). Detecting texts of arbitrary orientations in natural images. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.
70. Tieleman, T., and Hinton, G. (2012). Lecture 6. COURSERA: Neural Networks for Machine Learning, Coursera.
71. Bengio, Y. (2012). Practical recommendations for gradient-based training of deep architectures. Neural Networks: Tricks of the Trade, Springer.10.1007/978-3-642-35289-8_26 ↗
72. Liao, M., Wan, Z., Yao, C., Chen, K., and Bai, X. (2020, January 7–12). Real-time Scene Text Detection with Differentiable Binarization. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6812 ↗
73. OPMP: An Omnidirectional Pyramid Mask Proposal Network for Arbitrary-Shape Scene Text Detection10.1109/TMM.2020.2978630 ↗
74. TextMountain: Accurate scene text detection via instance segmentation10.1016/j.patcog.2020.107336 ↗
75. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature Pyramid Networks for Object Detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗