Real-Time Semantic Image Segmentation with Deep Learning for Autonomous Driving: A Survey
Source abstract
Abstract
Semantic image segmentation for autonomous driving is a challenging task due to its requirement for both effectiveness and efficiency. Recent developments in deep learning have demonstrated important performance boosting in terms of accuracy. In this paper, we present a comprehensive overview of the state-of-the-art semantic image segmentation methods using deep-learning techniques aiming to operate in real time so that can efficiently support an autonomous driving scenario. To this end, the presented overview puts a particular emphasis on the presentation of all those approaches which permit inference time reduction, while an analysis of the existing methods is addressed by taking into account their end-to-end functionality, as well as a comparative study that relies upon a consistent evaluation framework. Finally, a fruitful discussion is presented that provides key insights for the current trend and future research directions in real-time semantic image segmentation with deep learning for autonomous driving.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Visual Computing Group, Department of Electrical and Computer Engineering, Democritus University of Thrace, 67100 Xanthi, Greece
Department of Production and Management Engineering, Democritus University of Thrace, 67100 Xanthi, Greece
References (75)
Show source-provided references
1. Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art10.1561/0600000079 ↗
2. Wang, M., Liu, B., and Foroosh, H. (2017, January 22–29). Factorized Convolutional Neural Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.10.1109/ICCVW.2017.71 ↗
3. Guo, Y., Li, Y., Feris, R., Wang, L., and Simunic, T. (2019). Depthwise Convolution Is All You Need for Learning Multiple Visual Domains, AAAI.10.1609/aaai.v33i01.33018368 ↗
4. Chollet, F. (2017, January 21–26). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
5. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.E., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
6. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
7. Zhang, X., Zhou, X., Lin, M., and Sun, J. (2017). ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. arXiv.10.1109/CVPR.2018.00716 ↗
8. Paszke, A., Chaurasia, A., Kim, S., and Culurciello, E. (2016). ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation. arXiv.
9. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation10.1109/TPAMI.2016.2644615 ↗
10. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the Inception Architecture for Computer Vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
11. Fan, M., Lai, S., Huang, J., Wei, X., Chai, Z., Luo, J., and Wei, X. (2021). Rethinking BiSeNet For Real-time Semantic Segmentation. arXiv.10.1109/CVPR46437.2021.00959 ↗
12. Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
13. Nekrasov, V., Shen, C., and Reid, I. (2020). Template-Based Automatic Search of Compact Semantic Segmentation Architectures. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE.10.1109/WACV45572.2020.9093567 ↗
14. Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation, Springer International Publishing. Computer Vision–ECCV 2018.
15. Yu, C., Gao, C., Wang, J., Yu, G., Shen, C., and Sang, N. (2020). BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation. arXiv.10.1007/s11263-021-01515-2 ↗
16. Hong, Y., Pan, H., Sun, W., and Jia, Y. (2021). Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes. arXiv.
17. Verelst, T., and Tuytelaars, T. (2020). SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation. arXiv.10.1007/978-3-030-68238-5_2 ↗
18. Han, S., Pool, J., Tran, J., and Dally, W. (2015). Learning both Weights and Connections for Efficient Neural Network. arXiv.
19. Han, S., Mao, H., and Dally, W. (2016). Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. arXiv.
20. Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H.P. (2017). Pruning Filters for Efficient ConvNets. arXiv.
21. He, Y., Zhang, X., and Sun, J. (2017, January 22–29). Channel Pruning for Accelerating Very Deep Neural Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.155 ↗
22. Takos, G. (2020). A Survey on Deep Learning Methods for Semantic Image Segmentation in Real-Time. arXiv.
23. Wu, Z., Shen, C., and van den Hengel, A. (2017). Real-time Semantic Image Segmentation via Spatial Sparsity. arXiv.
24. Poudel, R.P.K., Bonde, U.D., Liwicki, S., and Zach, C. (2018). ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-Time, BMVC.
25. Poudel, R.P.K., Liwicki, S., and Cipolla, R. (2019). Fast-SCNN: Fast Semantic Segmentation Network. arXiv.
26. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–23). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
27. Chen, W., Gong, X., Liu, X., Zhang, Q., Li, Y., and Wang, Z. (2020, January 30). FasterSeg: Searching for Faster Real-time Semantic Segmentation. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
28. Neural Architecture Search: A Survey
29. Hutter, F., Kotthoff, L., and Vanschoren, J. (2019). Neural Architecture Search. Automated Machine Learning: Methods, Systems, Challenges, Springer International Publishing.10.1007/978-3-030-05318-5 ↗
30. Liu, C., Chen, L.C., Schroff, F., Adam, H., Hua, W., Yuille, A., and Li, F.-F. (2019, January 16–20). Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00017 ↗
31. Knowledge Distillation: A Survey10.1007/s11263-021-01453-z ↗
32. Wang, Y., Zhou, Q., and Wu, X. (2019). ESNet: An Efficient Symmetric Network for Real-Time Semantic Segmentation, PRCV.10.1007/978-3-030-31723-2_4 ↗
33. Zhuang, J., Yang, J., Gu, L., and Dvornek, N. (2019, January 27–28). ShelfNet for Fast Semantic Segmentation. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea.10.1109/ICCVW.2019.00113 ↗
34. Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). ICNet for Real-Time Semantic Segmentation on High-Resolution Images. Computer Vision—ECCV 2018, Springer International Publishing.
35. Bergman, A.W., and Lindell, D.B. (2019). Factorized Convolution Kernels in Image Processing, Stanford University.
36. Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation. Computer Vision—ECCV 2018, Springer International Publishing.
37. Mehta, S., Rastegari, M., Shapiro, L., and Hajishirzi, H. (2019, January 16–20). ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00941 ↗
38. Vallurupalli, N., Annamaneni, S., Varma, G., Jawahar, C.V., Mathew, M., and Nagori, S. (2018, January 18–22). Efficient Semantic Segmentation Using Gradual Grouping. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA.10.1109/CVPRW.2018.00102 ↗
39. Li, G., Yun, I., Kim, J., and Kim, J. (2019). DABNet: Depth-Wise Asymmetric Bottleneck for Real-Time Semantic Segmentation, BMVC.
40. Li, H., Xiong, P., Fan, H., and Sun, J. (2019, January 16–20). DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00975 ↗
41. Gamal, M., Siam, M., and Abdel-Razek, M. (2018). ShuffleSeg: Real-time Semantic Segmentation Network. arXiv.
42. Chao, P., Kao, C., Ruan, Y., Huang, C., and Lin, Y. (November, January 27). HarDNet: A Low Memory Traffic Network. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00365 ↗
43. ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation10.1109/TITS.2017.2750080 ↗
44. Wang, Y., Zhou, Q., Liu, J., Xiong, J., Gao, G., Wu, X., and Latecki, L. (2019, January 22–25). Lednet: A Lightweight Encoder-Decoder Network for Real-Time Semantic Segmentation. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803154 ↗
45. Lo, S.Y., Hang, H., Chan, S., and Lin, J.J. (2019, January 15–18). Efficient Dense Modules of Asymmetric Convolution for Real-Time Semantic Segmentation. Proceedings of the ACM Multimedia Asia, Beijing, China.10.1145/3338533.3366558 ↗
46. Nirkin, Y., Wolf, L., and Hassner, T. (2020). HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation. arXiv.10.1109/CVPR46437.2021.00405 ↗
47. Oršic, M., Krešo, I., Bevandic, P., and Šegvic, S. (2019, January 16–20). In Defense of Pre-Trained ImageNet Architectures for Real-Time Semantic Segmentation of Road-Driving Images. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01289 ↗
48. Hu, P., Caba, F., Wang, O., Lin, Z., Sclaroff, S., and Perazzi, F. (2020, January 13–19). Temporally Distributed Networks for Fast Video Semantic Segmentation. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00884 ↗
49. Treml, M., Arjona-Medina, J., Unterthiner, T., Durgesh, R., Friedmann, F., Schuberth, P., Mayr, A., Heusel, M., Hofmarcher, M., and Widrich, M. (2016, January 5–10). Speeding up Semantic Segmentation for Autonomous Driving. Proceedings of the 2016 Machine Learning for Intelligent Transportation Systems (MLITS) in Conjunction with the Thirtieth Conference on Neural Information Processing Systems (NIPS), Barcelona, Spain.
50. Chaurasia, A., and Culurciello, E. (2017, January 10–13). LinkNet: Exploiting encoder representations for efficient semantic segmentation. Proceedings of the 2017 IEEE Visual Communications and Image Processing (VCIP), St. Petersburg, FL, USA.10.1109/VCIP.2017.8305148 ↗
51. (2018). DSNet for Real-Time Driving Scene Semantic Segmentation. arXiv.
52. Nekrasov, V., Shen, C., and Reid, I.D. (2018). Light-Weight RefineNet for Real-Time Semantic Segmentation. arXiv.
53. Emara, T., Munim, H.E.A.E., and Abbas, H.M. (2019, January 2–4). LiteSeg: A Novel Lightweight ConvNet for Semantic Segmentation. Proceedings of the 2019 Digital Image Computing: Techniques and Applications (DICTA), Perth, Australia.10.1109/DICTA47822.2019.8945975 ↗
54. Berman, M., Triki, A., and Blaschko, M.B. (2018, January 18–23). The Lovasz-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00464 ↗
55. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The Cityscapes Dataset for Semantic Urban Scene Understanding. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
56. Semantic object classes in video: A high-definition ground truth database10.1016/j.patrec.2008.04.005 ↗
57. Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014). Microsoft COCO: Common Objects in Context. Computer Vision—ECCV 2014, Springer International Publishing.10.1007/978-3-319-10602-1 ↗
58. Vision meets robotics: The KITTI dataset10.1177/0278364913491297 ↗
59. Xie, J., Kiefel, M., Sun, M.T., and Geiger, A. (2016, January 27–30). Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.401 ↗
60. Ros, G., Sellart, L., Materzynska, J., Vazquez, D., and Lopez, A.M. (2016, January 27–30). The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.352 ↗
61. Neuhold, G., Ollmann, T., Bulò, S.R., and Kontschieder, P. (2017, January 22–29). The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.534 ↗
62. The ApolloScape Open Dataset for Autonomous Driving and Its Application10.1109/TPAMI.2019.2926463 ↗
63. Jin, J., Fatemi, A., Lira, W.P., Yu, F., Leng, B., Ma, R., Mahdavi-Amiri, A., and Zhang, H.R. (2021). RaidaR: A Rich Annotated Image Dataset of Rainy Street Scenes. arXiv.10.1109/ICCVW54120.2021.00330 ↗
64. A survey on deep learning techniques for image and video semantic segmentation10.1016/j.asoc.2018.05.018 ↗
65. Faniadis, E., and Amanatiadis, A. (2020, January 4–6). Deep Learning Inference at the Edge for Mobile and Aerial Robotics. Proceedings of the 2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Abu Dhabi, United Arab Emerites.10.1109/SSRR50563.2020.9292575 ↗
66. Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., and Maglogiannis, I. (2018). A Survey on Deep Transfer Learning. Artificial Neural Networks and Machine Learning—ICANN 2018, Springer International Publishing.10.1007/978-3-030-01418-6 ↗
67. Semantic Segmentation Algorithm Based on Attention Mechanism and Transfer Learning
68. Sharma, S., Ball, J.E., Tang, B., Carruth, D.W., Doude, M., and Islam, M.A. (2019). Semantic Segmentation with Transfer Learning for Off-Road Autonomous Driving. Sensors, 19.10.3390/s19112577 ↗
69. Csurka, G. (2020, January 1–4). Deep Visual Domain Adaptation. Proceedings of the 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), Timisoara, Romania.10.1109/SYNASC51798.2020.00013 ↗
70. Sankaranarayanan, S., Balaji, Y., Jain, A., Lim, S.N., and Chellappa, R. (2018, January 18–23). Learning from Synthetic Data: Addressing Domain Shift for Semantic Segmentation. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00395 ↗
71. Jing, L., and Tian, Y. (2020). Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2020.2992393 ↗
72. Deep clustering for weakly-supervised semantic segmentation in autonomous driving scenes10.1016/j.neucom.2019.11.019 ↗
73. Saleh, F.S., Aliakbarian, M.S., Salzmann, M., Petersson, L., and Alvarez, J.M. (2017, January 22–29). Bringing Background into the Foreground: Making All Classes Equal in Weakly-Supervised Video Semantic Segmentation. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.232 ↗
74. Strudel, R., Garcia, R., Laptev, I., and Schmid, C. (2021). Segmenter: Transformer for Semantic Segmentation. arXiv.10.1109/ICCV48922.2021.00717 ↗
75. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. (2017, January 21–26). Scene Parsing through ADE20K Dataset. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.544 ↗