Multi-Scale Feature Fusion with Attention Mechanism Based on CGAN Network for Infrared Image Colorization
Source abstract
Abstract
This paper proposes a colorization algorithm for infrared images based on a Conditional Generative Adversarial Network (CGAN) with multi-scale feature fusion and attention mechanisms, aiming to address issues such as color leakage and unclear semantics in existing infrared image coloring methods. Firstly, we improved the generator of the CGAN network by incorporating a multi-scale feature extraction module into the U-Net architecture to fuse features from different scales, thereby enhancing the network’s ability to extract features and improving its semantic understanding, which improves the problems of color leakage and blurriness during colorization. Secondly, we enhanced the discriminator of the CGAN network by introducing an attention mechanism module, which includes channel attention and spatial attention modules, to better distinguish between real and generated images, thereby improving the semantic clarity of the resulting infrared images. Finally, we jointly improved the generator and discriminator of the CGAN network by incorporating both the multi-scale feature fusion module and attention mechanism module. We tested our method on a dataset containing both infrared and near-infrared images, which retains more detailed features while also preserving the advantages of existing infrared images. The experimental results show that our proposed method achieved a peak signal-to-noise ratio (PSNR) of 16.5342 dB and a structural similarity index (SSIM) of 0.6385 on an RGB-NIR (Red, Green, Blue-Near Infrared) testing dataset, representing a 5% and 13% improvement over the original CGAN network, respectively. These results demonstrate the effectiveness of our proposed algorithm in addressing the issues of color leakage and unclear semantics in the original network. The proposed method in this paper is not only applicable to infrared image colorization but can also be widely applied to the colorization of remote sensing and CT images.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Fusion Techniques · Generative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
National Center for Materials Service Safety, University of Science and Technology Beijing, Beijing 100083, China
Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai), Zhuhai 519082, China
HBIS Materials Institute, No. 385 South Sports Street, Yuhua District, Shijiazhuang 050023, China
Hesteel Group Tangsteel Company, No. 9 Binhe Road, Tangshan 063000, China
Funding
HBIS Materials Institute · Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai)
References (31)
Show source-provided references
1. Shi, M., Zhang, J.Q., Chen, S.Y., Gao, L., Lai, Y.K., and Zhang, F.L. (2020). Deep Line Art Video Colorization with a Few References. arXiv.
2. Remote sensing image colorization using symmetrical multi-scale DCGAN in YUV color space10.1007/s00371-020-01933-2 ↗
3. Zheng, C.Y., Fu, Y., Zhao, Z., Wang, C., and Nie, J. (2020, January 13–15). Imbalance Satellite Image Colorization with Semantic Salience Priors. Proceedings of the Twelfth International Conference on Graphics and Image Processing (ICGIP 2020), Xi’an, China.10.1117/12.2589397 ↗
4. Khan, M., Gotoh, Y., and Nida, N. (2017, January 11–13). Medical image colorization for better visualization and segmentation. Proceedings of the Annual Conference on Medical Image Understanding and Analysis, MIUA 2017, Edinburgh, UK.10.1007/978-3-319-60964-5_50 ↗
5. Golyadkin, M., and Makarov, I. (2020, January 15–16). Semi-automatic Manga Colorization Using Conditional Adversarial Networks. Proceedings of the Analysis of Images, Social Networks and Texts: 9th International Conference, AIST 2020, Moscow, Russia.
6. Suarez, P.L., Sappa, A.D., Vintimilla, B.X., and Hammoud, R.I. (2018, January 7–10). Near InfraRed Imagery Colorization. Proceedings of the 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.10.1109/ICIP.2018.8451413 ↗
7. Application of near-infrared hyperspectral imaging for variety identification of coated maize kernels with deep learning10.1016/j.infrared.2020.103550 ↗
8. Application of medical infrared thermal imaging in the diagnosis of human internal focus-ScienceDirect10.1016/j.infrared.2019.06.013 ↗
9. Many-objective multilevel thresholding image segmentation for infrared images of power equipment with boost marine predators algorithm10.1016/j.asoc.2021.107905 ↗
10. IRT-GAN: A generative adversarial network with a multi-headed fusion strategy for automated defect detection in composites using infrared thermography10.1016/j.compstruct.2022.115543 ↗
11. Infrared dim target detection via mode-k1k2 extension tensor tubal rank under complex ocean environment10.1016/j.isprsjprs.2021.09.007 ↗
12. Gray-scale Image Colorization based on Conditional Deep Convolution Generation Adversarial Network
13. ISP-GAN: Inception sub-pixel deconvolution-based lightweight GANs for colorization10.1007/s11042-022-12587-8 ↗
14. A fully-automatic image colorization scheme using improved CycleGAN with skip connections10.1007/s11042-021-10881-5 ↗
15. Semantic-aware automatic image colorization via unpaired cycle-consistent self-supervised network10.1002/int.22667 ↗
16. Berg, A., Ahlberg, J., and Felsberg, M. (2018, January 18–22). Generating Visible Spectrum Images from Thermal Infrared. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA.10.1109/CVPRW.2018.00159 ↗
17. Nyberg, A., Eldesokey, A., Bergstrom, D., and Gustafsson, D. (2018, January 9). Unpaired Thermal to Visible Spectrum Transfer using Adversarial Training. Proceedings of the Multimodal Learning and Applications Workshop (MULA)-ECCV Workshop, Munich, Germany.10.1007/978-3-030-11024-6_49 ↗
18. Grayscale Enhancement Colorization Network for Visible-infrared Person Re-identification10.1109/TCSVT.2021.3072171 ↗
19. An improved DualGAN for near-infrared image colorization10.1016/j.infrared.2021.103764 ↗
20. A DenseUnet Generative Adversarial Network for Near-Infrared Face Image Colorization10.1016/j.sigpro.2021.108007 ↗
21. Thermal Infrared Image Colorization for Nighttime Driving Scenes with Top-Down Guided Attention10.1109/TITS.2022.3145476 ↗
22. AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks10.1109/TMM.2020.2997127 ↗
23. Learning a Deep Multi-Scale Feature Ensemble and an Edge-Attention Guidance for Image Fusion10.1109/TCSVT.2021.3056725 ↗
24. Mirza, M., and Osindero, S. (2014). Conditional Generative Adversarial Nets. Comput. Sci., 2672–2680.
25. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Networks. arXiv.
26. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
27. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2016, January 27–30). Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2017.632 ↗
28. Woo, S., Park, J., Lee, J.Y., and Kweon, I. (2018, January 8–14). CBAM: Convolutional Block Attention Module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
29. Brown, M., and Süsstrunk, S. (2011, January 21–23). Multi-spectral SIFT for scene category recognition. Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995637 ↗
30. An, Z., Yu, M., and Zhang, C. (2017, January 22–29). An improved Adam Algorithm using look-ahead. Proceedings of the 2017 International Conference, Venice, Italy.
31. MsRAN: A multi-scale residual attention network for multi-model image fusion10.1007/s11517-022-02690-1 ↗