journal article·2022-11-12Open access

TSSTNet: A Two-Stream Swin Transformer Network for Salient Object Detection of No-Service Rail Surface Defects

Chi Wan, Shuai Ma, Kechen Song

Coatings· Volume 12· Issue 11

Abstract

The detection of no-service rail surface defects is important in the rail manufacturing process. Detection of defects can prevent significant financial losses. However, the texture and form of the defects are often very similar to the background, which makes them difficult for the human eye to distinguish. How to accurately identify rail surface defects thus poses a challenge. We introduce salient object detection through machine vision to deal with this challenge. Salient object detection locates the most “significant” areas of an image using algorithms, which constitute an integral part of machine vision inspection. However, existing saliency detection networks suffer from inaccurate positioning, poor contouring, and incomplete detection. Therefore, we propose an innovative deep learning network named Two-Stream Swin Transformer Network (TSSTNet) for salient detection of no-service rail surface defects. Specifically, we propose a two-stream encoder—one stream for feature extraction and the other for edge extraction. TSSTNet also includes a three-stream decoder, consisting of a saliency stream, edge stream, and fusion stream. For the problem of incomplete detection, we innovatively introduce the Swin Transformer to model global information. For the problem of unclear contours, we expect to deepen the understanding of the difference in depth between the foreground and background through the learning of contour maps, so the contour alignment module (CAM) is created to deal with this problem. Moreover, to make the most of multimodal information, we suggest a multi-feature fusion module (MFFM). Finally, we conducted comparative experiments with 10 state-of-the-art (SOTA) approaches on the NRSD-MN datasets, and our model performed more competitively than others on five metrics.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Advanced Neural Network Applications · Visual Attention and Saliency Detection · Industrial Vision Systems and Defect Detection

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

School of Mechanical Engineering & Automation, Northeastern University, Shenyang 110819, China

National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Northeastern University, Shenyang 110819, China

Key Laboratory of Data Analytics and Optimization for Smart Industry, Ministry of Education, Shenyang 110819, China

Funding

National Natural Science Foundation of China · Fundamental Research Funds for the Central Universities · Central Guidance on Local Science and Technology Development Fund

References (51)

Show source-provided references
1. A hierarchical extractor-based visual rail surface inspection system10.1109/JSEN.2017.2761858 ↗
2. MCnet: Multiple Context Information Segmentation Network of No-Service Rail Surface Defects10.1109/TIM.2021.3127641 ↗
3. Research on defect detection of high-speed rail based on multi-frequency excitation composite electromagnetic method10.1016/j.measurement.2021.110351 ↗
4. Canny edge detection enhancement by general auto-regression model and bi-dimensional maximum conditional entropy10.1016/j.ijleo.2014.01.153 ↗
5. Defect detection of nickel plated punched steel strip based on improved least square method10.1016/j.ijleo.2020.164331 ↗
6. A novel rail defect detection method based on undecimated lifting wavelet packet transform and Shannon entropy-improved adaptive line enhancer10.1016/j.jsv.2018.04.003 ↗
7. Multi-scale Attention Guided Network for End-to-End Face Alignment and Recognition10.1016/j.jvcir.2022.103628 ↗
8. Spatial self-attention network with self-attention distillation for fine-grained image recognition10.1016/j.jvcir.2021.103368 ↗
9. Salient object detection: A survey10.1007/s41095-019-0149-9 ↗
10. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
11. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
12. Zhao, Z., Xia, C., Xie, C., and Li, J. (2021, January 20–24). Complementary trilateral decoder for fast and accurate salient object detection. Proceedings of the 29th ACM International Conference on Multimedia, Virtual.10.1145/3474085.3475494 ↗
13. Li, X., Yang, F., Cheng, H., and Liu, W. (2018, January 8–14). Contour knowledge transfer for salient object detection. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01267-0_22 ↗
14. Murugesan, B., Sarveswaran, K., Shankaranarayana, S.M., Ram, K., Joseph, J., and Sivaprakasam, M. (2019, January 23–27). Psi-Net: Shape and boundary aware joint multi-task deep network for medical image segmentation. Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany.10.1109/EMBC.2019.8857339 ↗
15. Edge-guided non-local fully convolutional network for salient object detection10.1109/TCSVT.2020.2980853 ↗
16. Surface Defect Classification of Steel Strip with Few Samples Based on Dual-Stream Neural Network10.1002/srin.202100554 ↗
17. Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., and Zhang, L. (2021, January 10–17). Cvt: Introducing convolutions to vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00009 ↗
18. Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8–14). Cbam: Convolutional block attention, module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
19. On the Canny edge detector10.1016/S0031-3203(00)00023-6 ↗
20. Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance10.3354/cr030079 ↗
21. Achanta, R., Hemami, S., Estrada, F., and Susstrunk, S. (2009, January 20–25). Frequency-tuned salient region detection. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPRW.2009.5206596 ↗
22. Fan, D.P., Cheng, M.M., Liu, Y., Li, T., and Borji, A. (2017, January 22–29). Structure-measure: A new way to evaluate foreground maps. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.487 ↗
23. Fan, D.P., Gong, C., Cao, Y., Ren, B., Cheng, M., and Borji, A. (2018). Enhanced-alignment measure for binary foreground map evaluation. arXiv.10.24963/ijcai.2018/97 ↗
24. Detectability of Rail Defects by Magnetic Flux Leakage Method10.1134/S1061830919040028 ↗
25. Detection of bearing/rail defects for linear motion stage using acoustic emission10.1007/s12541-013-0256-y ↗
26. Mehel-Saidi, Z., Bloch, G., and Aknin, P. (2008, January 25–29). A subspace method for detection and classification of rail defects. Proceedings of the 2008 16th European Signal Processing Conference, Lausanne, Switzerland.
27. Shi, H., Zhuang, L., Xu, X., Yu, Z., and Zhu, L. (2019). An Ultrasonic Guided Wave Mode Selection and Excitation Method in Rail Defect Detection. Appl. Sci., 9.10.3390/app9061170 ↗
28. MRSDI-CNN: Multi-model rail surface defect inspection system based on convolutional neural networks10.1109/TITS.2021.3101053 ↗
29. MtlrNet: An Effective Deep Multitask Learning Architecture for Rail Crack Detection
30. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
31. Zhao, J.X., Liu, J.J., Fan, D.P., and Cao, Y. (2019, January 27). EGNet: Edge guidance network for salient object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00887 ↗
32. Liu, N., Han, J., and Yang, M.H. (2018, January 18–23). Picanet: Learning pixel-wise contextual attention for saliency detection. Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00326 ↗
33. Zhao, T., and Wu, X. (2019, January 16–27). Pyramid feature attention network for saliency detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00320 ↗
34. U2-Net: Going deeper with nested U-structure for salient object detection10.1016/j.patcog.2020.107404 ↗
35. ASNet: An adaptive scale network for skin lesion segmentation in dermoscopy images
36. CAGNet: Content-aware guidance for salient object detection10.1016/j.patcog.2020.107303 ↗
37. Liu, J.J., Hou, Q., and Cheng, M.M. (2019). A simple pooling-based design for real-time salient object detection. arXiv.10.1109/CVPR.2019.00404 ↗
38. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
39. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30, Available online: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
40. Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F.E.H., Feng, J., and Yan, S. (2021, January 11–17). Tokens-to-token vit: Training vision transformers from scratch on imagenet. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00060 ↗
41. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. (2021, January 18–24). Training data-efficient image transformers & distillation through attention. Proceedings of the 38th International Conference on Machine Learning, PMLR 2021, Virtual.
42. Wang, W., Xie, E., Li, X., Fan, D., Song, K., Liang, D., Lu, T., and Luo, P. (2021, January 11–17). Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00061 ↗
43. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 18–20). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPR.2016.90 ↗
44. Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., and Jagersand, M. (2019, January 16–17). Basnet: Boundary-aware salient object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00766 ↗
45. Zhu, H., Li, P., Xie, H., Yan, X., Liang, D., Chen, D., Wei, M., and Qin, J. (2022, October 07). I can find you! Boundary-guided Separated Attention Network for Camouflaged Object Detection. AAAI 2022. Available online: https://ojs.aaai.org/index.php/AAAI/article/view/20273.10.1609/aaai.v36i3.20273 ↗
46. Sun, Y., Chen, G., Zhou, T., Zhang, Y., and Liu, N. (2021). Context-aware cross-level fusion network for camouflaged object detection. arXiv.10.24963/ijcai.2021/142 ↗
47. Wei, J., Wang, S., and Huang, Q. (2020, January 7–12). F3Net: Fusion, feedback and focus for salient object detection. Proceedings of the AAAI Conference on Artificial Intelligence 2020, New York, NY, USA.
48. Wang, B., Chen, Q., Zhou, M., Zhang, Z., Jin, X., and Gai, K. (2020, January 7–12). Progressive feature polishing network for salient object detection. Proceedings of the AAAI Conference on Artificial Intelligence 2020, New York, NY, USA.
49. Liu, J., Hou, Q., Liu, Z., and Cheng, M. (2022). PoolNet+: Exploring the Potential of Pooling for Salient Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE.10.1109/TPAMI.2021.3140168 ↗
50. Lee, M.S., Shin, W.S., and Han, S.W. (2022, January 24–28). TRACER: Extreme Attention Guided Salient Object Tracing Network (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence, Pomona, CA, USA.10.1609/aaai.v36i11.21633 ↗
51. Song, K., Wang, J., Bao, Y., Huang, L., and Yan, Y. (2022). A Novel Visible-Depth-Thermal Image Dataset of Salient Object Detection for Robotic Visual Perception. IEEE/ASME Transactions on Mechatronics, IEEE.10.1109/TMECH.2022.3215909 ↗