A Lightweight Network for Real-Time Rain Streaks and Rain Accumulation Removal from Single Images Captured by AVs
Source abstract
Abstract
In autonomous driving, object detection is considered a base step to many subsequent processes. However, object detection is challenged by loss in visibility caused by rain. Rainfall occurs in two main forms, which are streaks and streaks accumulations. Each degradation type imposes different effect on the captured videos; therefore, they cannot be mitigated in the same way. We propose a lightweight network which mitigates both types of rain degradation in real-time, without negatively affecting the object-detection task. The proposed network consists of two different modules which are used progressively. The first one is a progressive ResNet for rain streaks removal, while the second one is a transmission-guided lightweight network for rain streak accumulation removal. The network has been tested on synthetic and real rainy datasets and has been compared with state-of-the-art (SOTA) networks. Additionally, time performance evaluation has been performed to ensure real-time performance. Finally, the effect of the developed deraining network has been tested on YOLO object-detection network. The proposed network exceeded SOTA by 1.12 dB in PSNR on the average result of multiple synthetic datasets with 2.29× speedup. Finally, it can be observed that the inclusion of different lightweight stages works favorably for real-time applications and could be updated to mitigate different degradation factors such as snow and sun blare.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image Enhancement Techniques · Advanced Neural Network Applications · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Engineering, University of Central Lancashire, Preston PR1 2HE, UK
Arab Academy for Science, Technology, and Maritime Transport, Alexandria 1029, Egypt
References (70)
Show source-provided references
1. (2022, June 30). Road Traffic Injuries. Available online: https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries.
2. Automated driving recognition technologies for adverse weather conditions10.1016/j.iatssr.2019.11.005 ↗
3. SAE (2018). International: On-road automated vehicle standards committee and others. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, SAE International.
4. Object detection with deep learning: A review10.1109/TNNLS.2018.2876865 ↗
5. Vulnerable objects detection for autonomous driving: A review10.1016/j.vlsi.2021.01.002 ↗
6. Overview of environment perception for intelligent vehicles10.1109/TITS.2017.2658662 ↗
7. Autonomous vehicle perception: The technology of today and tomorrow10.1016/j.trc.2018.02.012 ↗
8. Vehicle detection techniques for collision avoidance systems: A review10.1109/TITS.2015.2409109 ↗
9. A review and an approach for object detection in images10.1504/IJCVR.2017.081234 ↗
10. A review of detection and tracking of object from image and video sequences
11. Zhiqiang, W., and Jun, L. (2017, January 26–28). A review of object detection based on convolutional neural network. Proceedings of the 2017 36th Chinese Control Conference (CCC), Dalian, China.10.23919/ChiCC.2017.8029130 ↗
12. Vision and rain10.1007/s11263-006-0028-6 ↗
13. Joint rain detection and removal from a single image with contextualized deep networks10.1109/TPAMI.2019.2895793 ↗
14. Ren, D., Zuo, W., Hu, Q., Zhu, P., and Meng, D. (2019, January 15–20). Progressive image deraining networks: A better and simpler baseline. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00406 ↗
15. Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., and Lau, R.W. (2019, January 15–20). Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01255 ↗
16. Wang, H., Wu, Y., Li, M., Zhao, Q., and Meng, D. (2019). A survey on rain removal from video and single image. arXiv.
17. Xu, J., Zhao, W., Liu, P., and Tang, X. (2012, January 25–27). Removing rain and snow in a single image using guided filter. Proceedings of the 2012 IEEE International Conference on Computer Science and Automation Engineering (CSAE), Zhangjiajie, China.10.1109/CSAE.2012.6272780 ↗
18. Zheng, X., Liao, Y., Guo, W., Fu, X., and Ding, X. (2013, January 5–10). Single-image-based rain and snow removal using multi-guided filter. Proceedings of the International Conference on Neural Information Processing, Lake Tahoe, NV, USA.10.1007/978-3-642-42051-1_33 ↗
19. Single image rain and snow removal via guided L0 smoothing filter10.1007/s11042-015-2657-7 ↗
20. Kim, J., Lee, C., Sim, J., and Kim, C. (2013, January 15–18). Single-image deraining using an adaptive nonlocal means filter. Proceedings of the 2013 IEEE International Conference on Image Processing, Melbourne, Australia.10.1109/ICIP.2013.6738189 ↗
21. Fu, Y., Kang, L., Lin, C., and Hsu, C. (2011, January 22–27). Single-frame-based rain removal via image decomposition. Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, Czech Republic.10.1109/ICASSP.2011.5946766 ↗
22. Visual depth guided color image rain streaks removal using sparse coding10.1109/TCSVT.2014.2308627 ↗
23. Kang, L., Lin, C., Lin, C., and Lin, Y. (2012, January 20–23). Self-learning-based rain streak removal for image/video. Proceedings of the 2012 IEEE International Symposium on Circuits and Systems (ISCAS), Seoul, Republic of Korea.10.1109/ISCAS.2012.6271635 ↗
24. A hierarchical approach for rain or snow removing in a single color image10.1109/TIP.2017.2708502 ↗
25. Sun, S., Fan, S., and Wang, Y.F. (2014, January 27–30). Exploiting image structural similarity for single image rain removal. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France.10.1109/ICIP.2014.7025909 ↗
26. Gu, S., Meng, D., Zuo, W., and Zhang, L. (2017, January 22–29). Joint convolutional analysis and synthesis sparse representation for single image layer separation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.189 ↗
27. Starck, J., Moudden, Y., Bobin, J., Elad, M., and Donoho, D.L. (August, January 31). Morphological component analysis, Wavelets XI. Proceedings of the Optics and Photonics Conference 2005, San Diego, CA, USA.10.1117/12.615237 ↗
28. Zhao, X., Liu, P., Liu, J., and Xianglong, T. (2008, January 24–27). The application of histogram on rain detection in video. Proceedings of the 11th Joint International Conference on Information Sciences, Khulna, Bangladesh.10.2991/jcis.2008.65 ↗
29. Single image deraining using deep convolutional networks10.1007/s11042-018-5825-8 ↗
30. Fastderainnet: A deep learning algorithm for single image deraining10.1109/ACCESS.2020.3008324 ↗
31. Video-based water drop detection and removal method for a moving vehicle10.3923/itj.2013.569.583 ↗
32. A rain pixel recovery algorithm for videos with highly dynamic scenes10.1109/TIP.2013.2290595 ↗
33. Clearing the skies: A deep network architecture for single-image rain removal10.1109/TIP.2017.2691802 ↗
34. Yang, W., Liu, J., and Feng, J. (2019, January 15–20). Frame-Consistent Recurrent Video Deraining With Dual-Level Flow. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00176 ↗
35. Li, M., Xie, Q., Zhao, Q., Wei, W., Gu, S., Tao, J., and Meng, D. (2018, January 18–23). Video rain streak removal by multiscale convolutional sparse coding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00695 ↗
36. Rain streak removal based on non-negative matrix factorization10.1007/s11042-017-5430-2 ↗
37. Chen, J., Tan, C., Hou, J., Chau, L., and Li, H. (2018, January 18–23). Robust video content alignment and compensation for rain removal in a cnn framework. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00658 ↗
38. Rain streak removal by multi-frame-based anisotropic filtering10.1007/s11042-015-3195-z ↗
39. Restoration of video by removing rain
40. You, S., Tan, R.T., Kawakami, R., and Ikeuchi, K. (2013, January 23–28). Adherent raindrop detection and removal in video. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.138 ↗
41. Ren, W., Tian, J., Han, Z., Chan, A., and Tang, Y. (2017, January 21–26). Video desnowing and deraining based on matrix decomposition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.303 ↗
42. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., and Paisley, J. (2017, January 21–26). Removing rain from single images via a deep detail network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.186 ↗
43. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
44. Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., and Jiang, J. (2020, January 13–19). Multi-scale progressive fusion network for single image deraining. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00837 ↗
45. Abdelhamed, A., Timofte, R., and Brown, M.S. (2019, January 16–17). Ntire 2019 challenge on real image denoising: Methods and results. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.
46. Deep learning on image denoising: An overview10.1016/j.neunet.2020.07.025 ↗
47. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising10.1109/TIP.2017.2662206 ↗
48. Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., and Yan, S. (2017, January 21–26). Deep joint rain detection and removal from a single image. Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA.10.1109/CVPR.2017.183 ↗
49. Yu, F., and Koltun, V. (2015). Multi-scale context aggregation by dilated convolutions. arXiv.
50. Zhang, H., and Patel, V.M. (2018, January 18–23). Density-aware single image de-raining using a multi-stream dense network. Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00079 ↗
51. Qian, R., Tan, R.T., Yang, W., Su, J., and Liu, J. (2018, January 18–23). Attentive generative adversarial network for raindrop removal from a single image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00263 ↗
52. Li, X., Wu, J., Lin, Z., Liu, H., and Zha, H. (2018, January 8–14). Recurrent squeeze-and-excitation context aggregation net for single image deraining. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_16 ↗
53. Lightweight pyramid networks for image deraining10.1109/TNNLS.2019.2926481 ↗
54. Fan, Z., Wu, H., Fu, X., Hunag, Y., and Ding, X. Residual-guide feature fusion network for single image deraining. arXiv, 2018.10.1145/3240508.3240694 ↗
55. Waqas Zamir, S., Arora, A., Khan, S., Hayat, M., Shahbaz Khan, F., Yang, M., and Shao, L. (2021). Multi-Stage Progressive Image Restoration. arXiv.10.1109/CVPR46437.2021.01458 ↗
56. Chen, L., Lu, X., Zhang, J., Chu, X., and Chen, C. (2021, January 20–25). HINet: Half instance normalization network for image restoration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPRW53098.2021.00027 ↗
57. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
58. Dehazenet: An end-to-end system for single image haze removal10.1109/TIP.2016.2598681 ↗
59. Hore, A., and Ziou, D. (2010, January 23–26). Image quality metrics: PSNR vs. SSIM. Proceedings of the 2010 20th International Conference on Pattern Recognition, Washington, DC, USA.10.1109/ICPR.2010.579 ↗
60. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
61. Making a “completely blind” image quality analyzer10.1109/LSP.2012.2227726 ↗
62. No-reference image quality assessment based on spatial and spectral entropies10.1016/j.image.2014.06.006 ↗
63. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel.
64. Vision and the atmosphere10.1023/A:1016328200723 ↗
65. Li, Z., Zhang, J., Zhong, R., Bhanu, B., Chen, Y., Zhang, Q., and Tang, H. (2021). Lightweight and Efficient Image Dehazing Network Guided by Transmission Estimation from Real-World Hazy Scenes. Sensors, 21.10.3390/s21030960 ↗
66. Mao, X., Shen, C., and Yang, Y. (2016). Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Advances in Neural Information Processing Systems, Curran Associates, Inc.
67. Glorot, X., and Bengio, Y. (2010, January 13–15). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, Sardinia, Italy.
68. Image de-raining using a conditional generative adversarial network10.1109/TCSVT.2019.2920407 ↗
69. Li, S., Araujo, I.B., Ren, W., Wang, Z., Tokuda, E.K., Junior, R.H., Cesar-Junior, R., Zhang, J., Guo, X., and Cao, X. (2019, January 15–20). Single image deraining: A comprehensive benchmark analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00396 ↗
70. Hnewa, M., and Radha, H. (2020). Object detection under rainy conditions for autonomous vehicles. arXiv.