Cascaded Dual-Inpainting Network for Scene Text
Source abstract
Abstract
Scene text inpainting is a significant research challenge in visual text processing, with critical applications spanning incomplete traffic sign comprehension, degraded container-code recognition, occluded vehicle license plate processing, and other incomplete scene text processing systems. In this paper, a cascaded dual-inpainting network for scene text (CDINST) is proposed. The architecture integrates two scene text inpainting models to reconstruct the text foreground: the Structure Generation Module (SGM) and Structure Reconstruction Module (SRM). The SGM primarily performs preliminary foreground text reconstruction and extracts text structures. Building upon the SGM’s guidance, the SRM subsequently enhances the foreground structure reconstruction through structure-guided refinement. The experimental results demonstrate compelling performance on the benchmark dataset, showcasing both the effectiveness of the proposed dual-inpainting network and its accuracy in incomplete scene text recognition. The proposed network achieves an average recognition accuracy improvement of 11.94% compared to baseline methods for incomplete scene text recognition tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Handwritten Text Recognition Techniques · Image Processing and 3D Reconstruction
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Tongji University, Shanghai 201804, China
Funding
Innovation Program of Shanghai Municipal Education Commission · National Natural Science Foundation of China
References (40)
Show source-provided references
1. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
2. Liu, H., Jiang, B., Xiao, Y., and Yang, C. (November, January 27). Coherent semantic attention for image inpainting. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
3. Guo, Z., Chen, Z., Yu, T., Chen, J., and Liu, S. (2019, January 21–25). Progressive image inpainting with full-resolution residual network. Proceedings of the 27th Acm International Conference on Multimedia, Nice, France.10.1145/3343031.3351022 ↗
4. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A.A. (2016, January 27–30). Context encoders: Feature learning by inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.278 ↗
5. Yan, Z., Li, X., Li, M., Zuo, W., and Shan, S. (2018, January 8–14). Shift-net: Image inpainting via deep feature rearrangement. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_1 ↗
6. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., and Catanzaro, B. (2018, January 8–14). Image inpainting for irregular holes using partial convolutions. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01252-6_6 ↗
7. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (November, January 27). Free-form image inpainting with gated convolution. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
8. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., and Bengio, Y. (2014). Generative adversarial nets. arXiv.
9. Globally and locally consistent image completion10.1145/3072959.3073659 ↗
10. Zeng, Y., Lin, Z., Lu, H., and Patel, V.M. (2021, January 10–17). Cr-fill: Generative image inpainting with auxiliary contextual reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01390 ↗
11. Yu, Y., Zhan, F., Lu, S., Pan, J., Ma, F., Xie, X., and Miao, C. (2021, January 10–17). Wavefill: A wavelet-based generation network for image inpainting. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01385 ↗
12. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. (2016, January 19–24). Generative adversarial text to image synthesis. Proceedings of the International Conference on Machine Learning, New York City, NY, USA.
13. Zhang, L., Chen, Q., Hu, B., and Jiang, S. (2020, January 12–16). Text-guided neural image inpainting. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.10.1145/3394171.3414017 ↗
14. Wu, X., Xie, Y., Zeng, J., Yang, Z., Yu, Y., Li, Q., and Liu, W. (2021, January 20–24). Adversarial learning with mask reconstruction for text-guided image inpainting. Proceedings of the 29th ACM International Conference on Multimedia, Chengdu, China.10.1145/3474085.3475506 ↗
15. MIGT: Multi-modal image inpainting guided with text10.1016/j.neucom.2022.11.074 ↗
16. Denoising diffusion probabilistic models
17. Nichol, A.Q., and Dhariwal, P. (, January 18–24). Improved denoising diffusion probabilistic models. Proceedings of the International Conference on Machine Learning, Vienna, Austria.
18. Photorealistic text-to-image diffusion models with deep language understanding
19. Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, S., Pellegrini, S., and Chan, W. (2023, January 18–22). Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01761 ↗
20. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022, January 18–24). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01042 ↗
21. Manukyan, H., Sargsyan, A., Atanyan, B., Wang, Z., Navasardyan, S., and Shi, H. (2023). HD-Painter: High-resolution and prompt-faithful text-guided image inpainting with diffusion models. arXiv.
22. Xie, S., Zhang, Z., Lin, Z., Hinz, T., and Zhang, K. (2023, January 18–22). Smartbrush: Text and shape guided object inpainting with diffusion model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.02148 ↗
23. Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Van Gool, L. (2022, January 18–24). Repaint: Inpainting using denoising diffusion probabilistic models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01117 ↗
24. Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen-Or, D. (2023, January 18–22). Null-text inversion for editing real images using guided diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00585 ↗
25. Ko, K., and Kim, C.S. (2023, January 2–6). Continuously masked transformer for image inpainting. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01211 ↗
26. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., and Timofte, R. (2021, January 11–17). Swinir: Image restoration using swin transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCVW54120.2021.00210 ↗
27. Jeevan, P., Kumar, D.S., and Sethi, A. (2023). WavePaint: Resource-Efficient Token-Mixer for Self-Supervised Inpainting. arXiv.
28. Xue, F., Zhang, J., Sun, J., Yin, J., Zou, L., and Li, J. (June, January 27). INIT: Inpainting Network for Incomplete Text. Proceedings of the 2022 IEEE International Symposium on Circuits and Systems (ISCAS2022), Austin, TX, USA.10.1109/ISCAS48785.2022.9937551 ↗
29. TSINIT: A Two-stage Inpainting Network for Incomplete Network10.1109/TMM.2022.3189245 ↗
30. Zhu, S., Fang, P., Zhu, C., Zhao, Z., Xu, Q., and Xue, H. (2024, January 20–27). Text image inpainting via global structure-guided diffusion models. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i7.28612 ↗
31. A Method of Inpainting Ancient Yi Characters Based on Dual Discriminator Generative Adversarial Networks
32. Fang, S., Xie, H., Wang, Y., Mao, Z., and Zhang, Y. (2021, January 19–25). Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00702 ↗
33. Aster: An attentional scene text recognizer with flexible rectification10.1109/TPAMI.2018.2848939 ↗
34. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition10.1109/TPAMI.2016.2646371 ↗
35. Yue, X., Kuang, Z., Lin, C., Sun, H., and Zhang, W. (2020, January 23–28). RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58529-7_9 ↗
36. Li, H., Wang, P., Shen, C., and Zhang, G. (February, January 27). Show, Attend and Read: A simple and strong baseline for irregular text recognition. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
37. Du, Y., Chen, Z., Jia, C., Yin, X., Zheng, T., Li, C., Du, Y., and Jiang, Y.-G. (2022, January 23–29). SVTR: Scene Text Recognition with a Single Visual Model. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence(IJCAI-22), Vienna, Austria.10.24963/ijcai.2022/124 ↗
38. Oktay, O., Schlemper, J., Le Folgoc, L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., and Kainz, B. (2018). Attention U-Net: Learning Where to Look for the Pancreas. arXiv.
39. Wang, Y., Chen, Y., Tao, X., and Jia, J. (2020, January 23–28). VCNet: A Robust Approach to Blind Image Inpainting. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK.10.1007/978-3-030-58595-2_45 ↗
40. A neural algorithm of artistic style10.1167/16.12.326 ↗