MCDCNet: Mask Classification Combined with Adaptive Dilated Convolution for Image Semantic Segmentation
Source abstract
Abstract
Effectively classifying each pixel in an image is an important research topic in semantic segmentation. The Existing methods typically require the network to directly generate a feature map of the same size as the original image and classify each pixel, which makes it difficult for the network to fully leverage the representations from the backbone. To handle this challenge, this paper proposes a method named mask classification combined with an adaptive dilated convolution network (MCDCNet). Firstly, a Vision Transformer (ViT)-based module is employed to capture contextual features as the backbone. Secondly, the Spatial Extraction Module (SEM) is proposed to extract multi-scale spatial information through adaptive dilated convolution while preserving the original feature size. This spatial information is then integrated into the corresponding contextual features to enhance the representation. Finally, a novel inference process is proposed that incorporates the instance activation map (IAM)-based decoder for semantic segmentation, thereby enhancing the network’s capability to capture and comprehend semantic features. The experimental results demonstrate that our network significantly outperforms other per-pixel classification networks across several semantic segmentation datasets. In particular, on Cityscapes, MCDCNet achieves 80.3 mIoU with 11.8 M Params, demonstrating that the network is able to deliver a strong segmentation performance while maintaining a relatively low parameter count.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: AI in cancer detection · Radiomics and Machine Learning in Medical Imaging · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Physics and Electronics, Nanning Normal University, Nanning 530100, China
References (50)
Show source-provided references
1. Fully Convolutional Networks for Semantic Segmentation10.1109/TPAMI.2016.2572683 ↗
2. Handwritten digit recognition with a back-propagation network
3. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid Scene Parsing Network. Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
4. Deep High-Resolution Representation Learning for Visual Recognition
5. Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. (2022, January 18–24). MetaFormer is Actually What You Need for Vision. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01055 ↗
6. Guo, M.H., Lu, C.Z., Hou, Q., Liu, Z.N., Cheng, M.M., and Min Hu, S. (December, January 28). SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA.
7. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. arXiv.
8. Cheng, T., Wang, X., Chen, S., Zhang, W., Zhang, Q., Huang, C., Zhang, Z., and Liu, W. (2022, January 18–24). Sparse Instance Activation for Real-Time Instance Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00439 ↗
9. Xu, J., Xiong, Z., and Bhattacharyya, S.P. (2023, January 17–24). PIDNet: A Real-time Semantic Segmentation Network Inspired from PID Controller. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01871 ↗
10. Per-Pixel Classification is Not All You Need for Semantic Segmentation
11. Jain, J., Li, J., Chiu, M., Hassani, A., Orlov, N., and Shi, H. (2023, January 17–24). OneFormer: One Transformer to Rule Universal Image Segmentation. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00292 ↗
12. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2020, January 16–20). Transformers: State-of-the-Art Natural Language Processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online.10.18653/v1/2020.emnlp-demos.6 ↗
13. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
14. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2014). Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. arXiv.
15. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2016). DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. arXiv.
16. Chen, L., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv.
17. Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., and Torr, P.H. (2021, January 19–25). Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers. Proceedings of the CVPR, Nashville, TN, USA.10.1109/CVPR46437.2021.00681 ↗
18. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
19. Xu, Z., Wu, D., Yu, C., Chu, X., Sang, N., and Gao, C. (2024, January 20–27). SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i6.28457 ↗
20. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.322 ↗
21. Chen, K., Pang, J., Wang, J., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Shi, J., and Ouyang, W. (2019, January 15–20). Hybrid Task Cascade for Instance Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00511 ↗
22. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-End Object Detection with Transformers. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
23. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., and Girdhar, R. (2022, January 18–24). Masked-attention Mask Transformer for Universal Image Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00135 ↗
24. Zhang, H., Li, F., Xu, H., Huang, S., Liu, S., Ni, L.M., and Zhang, L. (2023, January 17–24). MP-Former: Mask-Piloted Transformer for Image Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01733 ↗
25. Yu, F., and Koltun, V. (2017, January 21–26). Dilated Residual Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.75 ↗
26. Yu, F., and Koltun, V. (2016, January 2–4). Multi-Scale Context Aggregation by Dilated Convolutions. Proceedings of the International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
27. Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., and Cottrell, G.W. (2018, January 12–15). Understanding Convolution for Semantic Segmentation. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.10.1109/WACV.2018.00163 ↗
28. Wang, Z., and Ji, S. (2018, January 19–23). Smoothed Dilated Convolutions for Improved Dense Prediction. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3219944 ↗
29. Hanyu, T., Yamazaki, K., Tran, M., McCann, R.A., Liao, H., Rainwater, C., Adkins, M., Cothren, J., and Le, N. (2024). AerialFormer: Multi-Resolution Transformer for Aerial Image Segmentation. Remote Sens., 16.10.3390/rs16162930 ↗
30. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-Excitation Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
31. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
32. Lin, T.Y., Dollar, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature Pyramid Networks for Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
33. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal Loss for Dense Object Detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
34. Milletari, F., Navab, N., and Ahmadi, S.A. (2016, January 25–28). V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.10.1109/3DV.2016.79 ↗
35. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The Cityscapes Dataset for Semantic Urban Scene Understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
36. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. (2017, January 21–26). Scene parsing through ade20k dataset. Proceedings of the CVPR, Honolulu, HI, USA.10.1109/CVPR.2017.544 ↗
37. Caesar, H., Uijlings, J., and Ferrari, V. (2018, January 18–22). Coco-stuff: Thing and stuff classes in context. Proceedings of the CVPR, Salt Lake City, UT, USA.10.1109/CVPR.2018.00132 ↗
38. Contributors, M. (2024, August 01). MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark. Available online: https://github.com/open-mmlab/mmsegmentation.
39. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019, January 8–14). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
40. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F.F. (2009, January 20–25). ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of the CVPR 2009, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
41. Fan, M., Lai, S., Huang, J., Wei, X., Chai, Z., Luo, J., and Wei, X. (2021, January 20–25). Rethinking BiSeNet for Real-Time Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00959 ↗
42. Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., and Luo, P. (2021, January 6–14). SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. Proceedings of the Neural Information Processing Systems (NeurIPS), Online.
43. Zhang, W., Huang, Z., Luo, G., Chen, T., Wang, X., Liu, W., Yu, G., and Shen, C. (2022). TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation. arXiv.10.1109/CVPR52688.2022.01177 ↗
44. Zhang, Y., Wang, X., Wang, Y., Liu, Y., and Wang, X. (2023, January 1–5). SeaFormer: Squeeze-enhanced Axial Transformer for Mobile Semantic Segmentation. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
45. Bo, D., Pichao, W., and Wang, F. (2023, January 7–14). AFFormer: Head-Free Lightweight Semantic Segmentation with Linear Transformer. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.
46. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for MobileNetV3. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
47. Kirillov, A., He, K., Girshick, R., Rother, C., and Dollár, P. (2019, January 15–20). Panoptic Feature Pyramid Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00656 ↗
48. Liao, B., Wang, X., Zhu, L., Zhang, Q., and Huang, C. (2024). ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention. arXiv.
49. Yuan, Y., Fu, R., Huang, L., Lin, W., Zhang, C., Chen, X., and Wang, J. (2021). HRFormer: High-Resolution Transformer for Dense Prediction. arXiv.
50. Estimation for finite mixture of simplex models: Applications to biomedical data10.1177/1471082X17722607 ↗