An Efficient Module for Instance Segmentation Based on Multi-Level Features and Attention Mechanisms
Source abstract
Abstract
Recently, multi-level feature networks have been extensively used in instance segmentation. However, because not all features are beneficial to instance segmentation tasks, the performance of networks cannot be adequately improved by synthesizing multi-level convolutional features indiscriminately. In order to solve the problem, an attention-based feature pyramid module (AFPM) is proposed, which integrates the attention mechanism on the basis of a multi-level feature pyramid network to efficiently and pertinently extract the high-level semantic features and low-level spatial structure features; for instance, segmentation. Firstly, we adopt a convolutional block attention module (CBAM) into feature extraction, and sequentially generate attention maps which focus on instance-related features along the channel and spatial dimensions. Secondly, we build inter-dimensional dependencies through a convolutional triplet attention module (CTAM) in lateral attention connections, which is used to propagate a helpful semantic feature map and filter redundant informative features irrelevant to instance objects. Finally, we construct branches for feature enhancement to strengthen detailed information to boost the entire feature hierarchy of the network. The experimental results on the Cityscapes dataset manifest that the proposed module outperforms other excellent methods under different evaluation metrics and effectively upgrades the performance of the instance segmentation method.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Remote Sensing and LiDAR Applications · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
Funding
the National Natural Science Foundation of China · the Fundamental Research Funds for the Central Universities
References (44)
Show source-provided references
1. A survey on deep learning techniques for image and video semantic segmentation10.1016/j.asoc.2018.05.018 ↗
2. Chen, L.C., Hermans, A., Papandreou, G., Schroff, F., Wang, P., and Adam, H. (2018, January 18–22). MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00422 ↗
3. A survey on instance segmentation: State of the art10.1007/s13735-020-00195-x ↗
4. Chen, H., Sun, K., Tian, Z., Shen, C., Huang, Y., and Yan, Y. (2020, January 13–19). BlendMask: Top-down meets bottom-up for instance segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00860 ↗
5. Zagoruyko, S., Lerer, A., Lin, T.Y., Pinheiro, P.O., Gross, S., Chintala, S., and Dollár, P. (2016). A multipath network for object detection. arXiv.10.5244/C.30.15 ↗
6. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
7. Li, Y., Qi, H., Dai, J., Ji, X., and Wei, Y. (2017, January 21–26). Fully Convolutional Instance-Aware Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.472 ↗
8. Dai, J., He, K., and Sun, J. (2016, January 27–30). Instance-Aware Semantic Segmentation via Multi-task Network Cascades. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.343 ↗
9. Bai, M., and Urtasun, R. (2017, January 21–26). Deep Watershed Transform for Instance Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.305 ↗
10. Neven, D., Brabandere, B.D., Proesmans, M., and Gool, L.V. (2019, January 15–20). Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00904 ↗
11. Arnab, A., and Torr, P.H. (2017, January 21–26). Pixelwise Instance Segmentation with a Dynamically Instantiated Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.100 ↗
12. Bolya, D., Zhou, C., Xiao, F., and Lee, Y. (2019, January 27–28). YOLACT: Real-time instance segmentation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00925 ↗
13. Sofiiuk, K., Barinova, O., and Konushin, A. (2019, January 27–28). Adaptis: Adaptive instance selection network. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00745 ↗
14. Liu, S., Qi, L., Qin, H., Shi, J., and Jia, J. (2018, January 18–22). Path aggregation network for instance segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00913 ↗
15. Lin, T., Dollar, P., Girshick, R., He, K., Hariharan, B., and Belongie, S.J. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
16. Hu, J., Shen, L., and Sun, G. (2018, January 18–22). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
17. Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., and Tang, X. (2017, January 21–26). Residual attention network for image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.683 ↗
18. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–22). Non-local neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
19. Chen, Y., Kalantidis, Y., Li, J., Yan, S., and Feng, J. (2018, January 3–8). A2-nets: Double attention networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
20. Woo, S., Park, J., Lee, J., and Kweon, I. (2018, January 8–14). Cbam: Convolutional block attention module. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
21. Misra, D., Nalamada, T., Arasanipalai, A., and Hou, Q. (2020). Rotate to Attend: Convolutional Triplet Attention Module. arXiv.10.1109/WACV48630.2021.00318 ↗
22. Faster R-CNN: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
23. Liu, S., Jia, J., Fidler, S., and Urtasun, R. (2017, January 22–29). Sequential grouping networks for instance segmentation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.378 ↗
24. Gao, N., Shan, Y., Wang, Y., Zhao, X., Yu, Y., Yang, M., and Huang, K. (2019, January 27–28). Ssap: Single-shot instance segmentation with affinity pyramid. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00073 ↗
25. Wang, X., Kong, T., Shen, C., Jiang, Y., and Li, L. (2019). SOLO: Segmenting objects by locations. arXiv.10.1007/978-3-030-58523-5_38 ↗
26. Xie, E., Sun, P., Song, X., Wang, W., Liu, X., Liang, D., Shen, C., and Luo, P. (2020, January 16–20). PolarMask: Single shot instance segmentation with polar representation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01221 ↗
27. Wang, X., Zhang, R., Kong, T., Li, L., and Shen, C. (2020). SOLOv2: Dynamic and Fast Instance Segmentation. arXiv.
28. Zhang, J., Yan, Y., Cheng, Z., and Wang, W. (2020). Lightweight Attention Pyramid Network for Object Detection and Instance Segmentation. Appl. Sci., 10.10.3390/app10030883 ↗
29. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
30. Bell, S., Zitnick, C.L., Bala, K., and Girshick, R.B. (2016, January 27–30). Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.314 ↗
31. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-Net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
32. Shrivastava, A., Sukthankar, R., Malik, J., and Gupta, A. (2016). Beyond skip connections: Top-down modulation for object detection. arXiv.
33. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C., and Berg, A. (2016, January 8–16). SSD: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
34. Cao, Y., Xu, J., Lin, S., Wei, F., and Hu, H. (2019, January 27–28). Gcnet: Non-local networks meet squeeze-excitation networks and beyond. Proceedings of the IEEE International Conference on Computer Vision Workshops, Seoul, Korea.10.1109/ICCVW.2019.00246 ↗
35. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
36. Zeiler, D., and Fergus, R. (2014, January 6–12). Visualizing and understanding convolutional networks. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10590-1_53 ↗
37. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019, January 8–14). Pytorch: An imperative style, high-performance deep learning library. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
38. Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., and Xu, J. (2019). MMDetection: Open mmlab detection toolbox and benchmark. arXiv.
39. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The Cityscapes Dataset for Semantic Urban Scene Understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
40. Acuna, D., Ling, H., Kar, A., and Fidler, S. (2018, January 18–22). Efficient interactive annotation of segmentation datasets with polygon-rnn++. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00096 ↗
41. Liu, Y., Yang, S., Li, B., Zhou, W., Xu, J., Li, H., and Lu, Y. (2018, January 8–14). Affinity derivation and graph merge for instance segmentation. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01219-9_42 ↗
42. Uhrig, J., Rehder, E., Frohlich, B., Franke, U., and Brox, T. (2018, January 26–29). Box2pix: Single-shot instance segmentation by assigning pixels to object boxes. Proceedings of the IEEE Intelligent Vehicles Symposium, SuZhou, China.10.1109/IVS.2018.8500621 ↗
43. Hayder, Z., He, X., and Salzmann, M. (2017, January 21–26). Boundary-aware instance segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.70 ↗
44. De Brabandere, B., Neven, D., and van Gool, L. (2017). Semantic instance segmentation with a discriminative loss function. arXiv.10.1109/CVPRW.2017.66 ↗