LightSeg: Local Spatial Perception Convolution for Real-Time Semantic Segmentation
Source abstract
Abstract
Semantic segmentation is increasingly being applied on mobile devices due to advancements in mobile chipsets, particularly in low-power consumption scenarios. However, the lightweight design of mobile devices poses limitations on the receptive field, which is crucial for dense prediction problems. Existing approaches have attempted to balance lightweight designs and high accuracy by downsampling features in the backbone. However, this downsampling may result in the loss of local details at each network stage. To address this challenge, this paper presents a novel solution in the form of a compact and efficient convolutional neural network (CNN) for real-time applications: our proposed model, local spatial perception convolution (LSPConv). Furthermore, the effectiveness of our architecture is demonstrated on the Cityscapes dataset. The results show that our model achieves an impressive balance between accuracy and inference speed. Specifically, our LightSeg, which does not rely on ImageNet pretraining, achieves an mIoU of 76.1 at a speed of 61 FPS on the Cityscapes validation set, utilizing an RTX 2080Ti GPU with mixed precision. Additionally, it achieves a speed of 115.7 FPS on the Jetson NX with int8 precision.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Visual Attention and Saliency Detection · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China
Guangxi Key Laboratory of Image and Graphic Intelligent Processing, Guilin University of Electronic Technology, Guilin 541004, China
Funding
National Natural Science Foundation of China · Nature Science Key Foundation of Guangxi · Sichuan Regional Innovation Cooperation Project · Guangxi Key Laboratory of Image and Graphic Intelligent Processing · Student’s Platform for Innovation and Entrepreneurship Training Program
References (41)
Show source-provided references
1. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.
2. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L. (2018, January 18–22). Mobilenetv2: The next generation of on-device computer vision networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.
3. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for mobilenetv3. Proceedings of the IEEE International Conference on Computer Vision, Seoul, South Korea.
4. Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 18–22). Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00716 ↗
5. Intelligent generation of Peking opera facial masks with deep learning frameworks10.1186/s40494-023-00865-z ↗
6. Zhu, Z., Xu, M., Bai, S., Huang, T., and Bai, X. (November, January 27). Asymmetric non-local neural networks for semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, South Korea.
7. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 16–20). Dual attention network for scene segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00326 ↗
8. A semantic and emotion-based dual latent variable generation model for a dialogue system10.1049/cit2.12153 ↗
9. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. (2017, January 21–26). Aggregated residual transformations for deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.634 ↗
10. Gao, R. (2023, January 18–22). Rethinking Dilated Convolution for Real-Time Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Vancouver, BC, Canada.10.1109/CVPRW59228.2023.00493 ↗
11. Yu, F., Koltun, V., and Funkhouser, T. (2017, January 21–26). Dilated residual networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.75 ↗
12. Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., and Cottrell, G. (2018, January 12–15). Understanding convolution for semantic segmentation. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.10.1109/WACV.2018.00163 ↗
13. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
14. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, Proceedings of the 18th International Conference, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
15. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2014). Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv.
16. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
17. Chen, L.C., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. arXiv.
18. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
19. Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., and Torr, P.H. (2021, January 19–25). Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.00681 ↗
20. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., and Zhou, Y. (2021). Transunet: Transformers make strong encoders for medical image segmentation. arXiv.
21. SegFormer: Simple and efficient design for semantic segmentation with transformers
22. LRD-SLAM: A Lightweight Robust Dynamic SLAM Method by Semantic Segmentation Network10.1155/2022/7332390 ↗
23. Maaz, M., Shaker, A., Cholakkal, H., Khan, S., Zamir, S.W., Anwer, R.M., and Shahbaz Khan, F. (2022). European Conference on Computer Vision, Springer.
24. Chen, Y., Dai, X., Chen, D., Liu, M., Dong, X., Yuan, L., and Liu, Z. (2022, January 19–24). Mobile-former: Bridging mobilenet and transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00520 ↗
25. Impact of data processing on deriving micro-mobility patterns from vehicle availability data10.1016/j.trd.2021.102913 ↗
26. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
27. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The Cityscapes Dataset for Semantic Urban Scene Understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
28. Brostow, G.J., Shotton, J., Fauqueur, J., and Cipolla, R. (2008, January 12–18). Segmentation and Recognition Using Structure from Motion Point Clouds. Proceedings of the European Conference on Computer Vision (ECCV), Marseille, France.10.1007/978-3-540-88682-2_5 ↗
29. Semantic Object Classes in Video: A High-Definition Ground Truth Database10.1016/j.patrec.2008.04.005 ↗
30. Cubuk, E.D., Zoph, B., Shlens, J., and Le, Q.V. (2020, January 13–19). Randaugment: Practical automated data augmentation with a reduced search space. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, hlSeattle, WA, USA.10.1109/CVPRW50498.2020.00359 ↗
31. Zhu, Y., Sapra, K., Reda, F.A., Shih, K.J., Newsam, S., Tao, A., and Catanzaro, B. (2019, January 16–20). Improving semantic segmentation via video propagation and label relaxation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00906 ↗
32. Si, H., Zhang, Z., Lv, F., Yu, G., and Lu, F. (2019). Real-time semantic segmentation via multiply spatial fusion network. arXiv.
33. Orsic, M., Kreso, I., Bevandic, P., and Segvic, S. (2019, January 16–20). In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01289 ↗
34. Chen, W., Gong, X., Liu, X., Zhang, Q., Li, Y., and Wang, Z. (2019). FasterSeg: Searching for Faster Real-time Semantic Segmentation. arXiv.
35. Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., and Sang, N. (2018, January 8–14). Bisenet: Bilateral segmentation network for real-time semantic segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01261-8_20 ↗
36. Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes10.1109/TITS.2022.3228042 ↗
37. Fan, M., Lai, S., Huang, J., Wei, X., Chai, Z., Luo, J., and Wei, X. (2021, January 19–25). Rethinking bisenet for real-time semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.00959 ↗
38. Lin, P., Sun, P., Cheng, G., Xie, S., Li, X., and Shi, J. (2020, January 13–19). Graph-guided architecture search for real-time semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, hlSeattle, WA, USA.10.1109/CVPR42600.2020.00426 ↗
39. Zhang, Y., Qiu, Z., Liu, J., Yao, T., Liu, D., and Mei, T. (2019, January 16–20). Customizable architecture search for semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01191 ↗
40. Chandra, S., Couprie, C., and Kokkinos, I. (2018, January 18–22). Deep spatio-temporal random fields for efficient video segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00929 ↗
41. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation10.1007/s11263-021-01515-2 ↗