Synthetic Data-Driven Real-Time Detection Transformer Object Detection in Raining Weather Conditions
Source abstract
Abstract
Images captured in rainy weather conditions often suffer from contamination, resulting in blurred or obscured objects, which can significantly impact detection performance due to the loss of identifiable texture and color information. Moreover, the quality of the detection model plays a pivotal role in determining detection outcomes. This study adopts a dual perspective, considering both pre-trained models and training data. It employs 15 image augmentation techniques, combined with neural style transfer (NST), CycleGAN, and an analytical method, to synthesize images under rainy conditions. The Real-Time Detection Transformer (RTDETR) and YOLOv8 pre-trained models are utilized to establish object detection frameworks tailored for rainy weather conditions. Testing is carried out using the DAWN (Detection in Adverse Weather Nature) dataset. The findings suggest compatibility between the pre-trained detection models and various data synthesis methods. Notably, YOLOv8 exhibits better compatibility with CycleGAN data synthesis, while RTDETR demonstrates a stronger alignment with the NST and analytical approaches. Upon the integration of synthesized rainy images into model training, RTDETR demonstrates significantly enhanced detection accuracy compared to YOLOv8, indicating a more pronounced improvement in performance. The proposed approach of combining RTDETR with NST in this study shows a significant improvement in Recall (R) and mAP50-95 by 16.35% and 15.50%, respectively, demonstrating the robust rainy weather resilience of this method. Additionally, RTDETR outperforms YOLOv8 in terms of inference speed and hardware requirements, making it easier to use and deploy in real-time applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image Enhancement Techniques · Fire Detection and Safety Systems · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
GIS Research Center, Feng Chia University, Taichung 40724, Taiwan
Ph.D. Program for Infrastructure Planning and Engineering, Feng Chia University, Taichung 40724, Taiwan
Department of Land Economics, National Chengchi University, Taipei 11605, Taiwan
References (54)
Show source-provided references
1. Fast and accurate object detector for autonomous driving based on improved YOLOv510.1038/s41598-023-36868-w ↗
2. MCS-YOLO: A multiscale object detection method for autonomous driving road environment recognition10.1109/ACCESS.2023.3252021 ↗
3. Deep learning-based image 3-d object detection for autonomous driving10.1109/JSEN.2023.3235830 ↗
4. Fighting against terrorism: A real-time CCTV autonomous weapons detection based on improved YOLO v410.1016/j.dsp.2022.103790 ↗
5. Development of early fire detection model for buildings using computer vision-based CCTV10.1016/j.jobe.2022.105647 ↗
6. Seabird surveillance: Combining CCTV and artificial intelligence for monitoring and research10.1002/rse2.329 ↗
7. Zhu, J.-Y., Park, T., Isola, P., and Efros, A.A. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. arXiv.10.1109/ICCV.2017.244 ↗
8. Porav, H., Maddern, W., and Newman, P. (2018, January 21–25). Adversarial Training for Adverse Conditions: Robust Metric Localisation Using Appearance Transfer. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.10.1109/ICRA.2018.8462894 ↗
9. Teeti, I., Musat, V., Khan, S., Rast, A., Cuzzolin, F., and Bradley, A. (2022). Vision in adverse weather: Augmentation using CycleGANs with various object detectors for robust perception in autonomous racing. arXiv.
10. Musat, V., Fursa, I., Newman, P., Cuzzolin, F., and Bradley, A. (2021, January 11–17). Multi-weather city: Adverse weather stacking for autonomous driving. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada.10.1109/ICCVW54120.2021.00325 ↗
11. Uricar, M., Krizek, P., Sistu, G., and Yogamani, S. (2019). SoilingNet: Soiling Detection on Automotive Surround-View Cameras. arXiv.10.1109/ITSC.2019.8917178 ↗
12. Rothmeier, T., and Huber, W. (2021, January 19–22). Let it Snow: On the Synthesis of Adverse Weather Image Data. Proceedings of the 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA.10.1109/ITSC48978.2021.9565008 ↗
13. Development of robust detector using the weather deep generative model for outdoor monitoring system10.1016/j.eswa.2023.120984 ↗
14. Chang, K.-C., Liao, Y.-D., and Kuo, C.-Y. (2022, January 6–8). Enhancing Recognition Accuracy of Urban Flooding by Generating Synthetic Samples Using CycleGAN. Proceedings of the 2022 IEEE International Conference on Consumer Electronics—Taiwan, Taipei, Taiwan.10.1109/ICCE-Taiwan55306.2022.9869172 ↗
15. Son, D.-M., Kwon, H.-J., and Lee, S.-H. (2023). Enhanced Night-to-Day Image Conversion Using CycleGAN-Based Base-Detail Paired Training. Mathematics, 11.10.3390/math11143102 ↗
16. Chen, K., Li, H., Li, C., Zhao, X., Wu, S., Duan, Y., and Wang, J. (2022). An Automatic Defect Detection System for Petrochemical Pipeline Based on Cycle-GAN and YOLO v5. Sensors, 22.10.3390/s22207907 ↗
17. ColorAI-automatic image Colorization using CycleGAN
18. Gatys, L.A., Ecker, A.S., and Bethge, M. (2016, January 27–30). Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.265 ↗
19. Jing, Y., Yang, Y., Feng, Z., Ye, J., Yu, Y., and Song, M. (2017). Neural Style Transfer: A Review. arXiv.
20. Xu, Y., and Goel, A. (2019). Cross-Domain Image Classification through Neural-Style Transfer Data Augmentation. arXiv.
21. Cygert, S., and Czyzewski, A. (2019, January 18–20). Style Transfer for Detecting Vehicles with Thermal Camera. Proceedings of the 2019 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), Poznan, Poland.10.23919/SPA.2019.8936707 ↗
22. Ding, Y., and Luo, X. (2023, January 25–28). Personal Protective Equipment Detection in Extreme Construction Conditions. Proceedings of the Computing in Civil Engineering 2023, Corvallis, OR, USA.10.1061/9780784485248.081 ↗
23. Fan, Y., Wang, Y., Wei, M., Wang, F.L., and Xie, H. (2024). FriendNet: Detection-Friendly Dehazing Network. arXiv.
24. Narasimhan, S.G., and Nayar, S.K. (2000, January 15). Chromatic framework for vision in bad weather. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), Hilton Head, SC, USA.
25. Single image haze removal using dark channel prior
26. Guo, C., Li, C., Guo, J., Loy, C.C., Hou, J., Kwong, S., and Cong, R. (2020, January 13–19). Zero-reference deep curve estimation for low-light image enhancement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00185 ↗
27. Qian, R., Tan, R.T., Yang, W., Su, J., and Liu, J. (2018, January 18–23). Attentive generative adversarial network for raindrop removal from a single image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00263 ↗
28. Single image rain removal via a deep decomposition–composition network10.1016/j.cviu.2019.05.003 ↗
29. Liu, W., Ren, G., Yu, R., Guo, S., Zhu, J., and Zhang, L. (March, January 22). Image-adaptive YOLO for object detection in adverse weather conditions. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.
30. Zhang, K., Yan, X., Wang, Y., and Qi, J. (2023). International Conference on Artificial Neural Networks, Springer.
31. Togethernet: Bridging image restoration and object detection together via dynamic enhancement learning10.1111/cgf.14692 ↗
32. Wang, K., Wang, T., Qu, J., Jiang, H., Li, Q., and Chang, L. (2022). An End-to-End Cascaded Image Deraining and Object Detection Neural Network. arXiv.10.1109/LRA.2022.3192200 ↗
33. Qiu, Y., Lu, Y., Wang, Y., and Jiang, H. (2023). IDOD-YOLOV7: Image-dehazing YOLOV7 for object detection in low-light foggy traffic environments. Sensors, 23.10.3390/s23031347 ↗
34. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
35. Redmon, J., and Farhadi, A. (2017, January 21–26). YOLO9000: Better, faster, stronger. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.690 ↗
36. Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv.
37. Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv.
38. Wang, C.-Y., Bochkovskiy, A., and Liao, H.-Y.M. (2023, January 17–24). YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00721 ↗
39. Wang, C.-Y., Yeh, I.-H., and Liao, H.-Y.M. (2024). YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information. arXiv.
40. Jocher, G. (2023, May 02). YOLOv5 by Ultralytics. Available online: https://github.com/ultralytics/yolov5.
41. Jocher, G., Chaurasia, A., and Qiu, J. (2023, May 02). YOLO by Ultralytics. Available online: https://github.com/ultralytics/ultralytics.
42. Li, C., Li, L., Jiang, H., Weng, K., Geng, Y., Li, L., Ke, Z., Li, Q., Cheng, M., and Nie, W. (2022). YOLOv6: A single-stage object detection framework for industrial applications. arXiv.
43. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
44. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020). European Conference on Computer Vision–ECCV 2020, Springer.
45. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2020). Deformable detr: Deformable transformers for end-to-end object detection. arXiv.
46. Meng, D., Chen, X., Fan, Z., Zeng, G., Li, H., Yuan, Y., Sun, L., and Wang, J. (2021, January 10–17). Conditional detr for fast training convergence. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00363 ↗
47. Lv, W., Xu, S., Zhao, Y., Wang, G., Wei, J., Cui, C., Du, Y., Dang, Q., and Liu, Y. (2023). Detrs beat yolos on real-time object detection. arXiv.
48. Volk, G., Müller, S., Von Bernuth, A., Hospach, D., and Bringmann, O. (2019, January 27–30). Towards robust CNN-based object detection through augmentation with synthetic rain variations. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.10.1109/ITSC.2019.8917269 ↗
49. Tremblay, M., Halder, S.S., de Charette, R., and Lalonde, J.-F. (2020). Rain rendering for evaluating and improving robustness to bad weather. arXiv.10.1007/s11263-020-01366-3 ↗
50. Weather-aware object detection method for maritime surveillance systems10.1016/j.future.2023.09.030 ↗
51. Gupta, H., Kotlyar, O., Andreasson, H., and Lilienthal, A.J. (2024, January 4–8). Robust Object Detection in Challenging Weather Conditions. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV57701.2024.00735 ↗
52. Boreas: A multi-season autonomous driving dataset10.1177/02783649231160195 ↗
53. Kenk, M.A., and Hassaballah, M. (2020). DAWN: Vehicle Detection in Adverse Weather Nature Dataset. arXiv.
54. Sun, P., Zhang, R., Jiang, Y., Kong, T., Xu, C., Zhan, W., and Luo, P. (2021, January 20–25). Sparse r-cnn: End-to-end object detection with learnable proposals. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01422 ↗