Near-Infrared Image Colorization Using Asymmetric Codec and Pixel-Level Fusion
Source abstract
Abstract
This paper mainly studies the colorization of near-infrared (NIR) images. Image colorization methods cannot be extended to NIR image colorization since the wavelength band of the NIR image exceeds the visible light spectral range and it is often linearly independent of the luminance of the RGB image. Furthermore, a symmetric codec, which cannot guarantee the ability of the encoder to extract features, is often used as the main frame of the network in both CNN-based colorization networks and CycleGAN-based colorization networks. In order to deal with the investigated problem, we propose a novel NIR colorization method using asymmetric codec (ACD) and pixel-level fusion. ACD is designed to improve the feature extraction ability of the encoder by allowing the information to enter deeper into the model and learning more non-redundant information. In addition, the global and local feature fusion networks (GLFFNet) are embedded between the encoder and the decoder to improve the prediction of the subtle color information of the image. The ACD and GLFFNet together constitute the colorization network (ColorNet) in this paper. Bilateral filtering and weighted least squares filtering (BFWLS) are used to fuse the pixel-level information of the input NIR image into the raw output image of the ColorNet. Finally, an intensive comparison analysis based on common datasets is conducted to verify superiority over existing methods in qualitative and quantitative visual assessments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Fusion Techniques · Image Enhancement Techniques · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
China School of Communication and Information Engineering, Shanghai University, Shanghai 200444, China
Funding
the Innovation Action Plan of Shanghai Science and Technology Commission
References (41)
Show source-provided references
1. Luo, Y., Remillard, J., and Hoetzer, D. (2010, January 21–24). Pedestrian Detection in Near-Infrared Night Vision System. Proceedings of the 2010 IEEE Intelligent Vehicles Symposium, IEEE, La Jolla, CA, USA.10.1109/IVS.2010.5548089 ↗
2. Near-infrared camera for night surveillance applications
3. Near infrared spectroscopy in wildlife and biodiversity10.1255/jnirs.1199 ↗
4. Denoising and fusion method of night vision image based on wavelet transform
5. Limmer, M., and Lensch, H.P. (2016, January 18–20). Infrared Colorization Using Deep Convolutional Neural Networks. Proceedings of the 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE, Anaheim, CA, USA.10.1109/ICMLA.2016.0019 ↗
6. Dong, Z., Kamata, S.-I., and Breckon, T.P. (2018, January 7–10). Infrared Image Colorization Using a S-Shape Network. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), IEEE, Athens, Greece.10.1109/ICIP.2018.8451230 ↗
7. Fredembach, C., and Süsstrunk, S. (2008, January 9–13). Colouring the Near-Infrared. Proceedings of the Color and Imaging Conference, Society for Imaging Science and Technology, Terrassa, Spain.
8. Chitu, M. (2021, January 2–3). Near-Infrared Colorization using Neural Networks for In-Cabin Enhanced Video Conferencing. Proceedings of the 2021 International Aegean Conference on Electrical Machines and Power Electronics (ACEMP) & 2021 International Conference on Optimization of Electrical and Electronic Equipment (OPTIM), IEEE, Brasov, Romania.10.1109/OPTIM-ACEMP50812.2021.9590054 ↗
9. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
10. Nir to rgb domain translation using asymmetric cycle generative adversarial networks10.1109/ACCESS.2019.2933671 ↗
11. Chaurasia, A., and Culurciello, E. (2017, January 10–13). Linknet: Exploiting Encoder Representations for Efficient Semantic Segmentation. Proceedings of the 2017 IEEE Visual Communications and Image Processing (VCIP), IEEE, St. Petersburg, FL, USA.10.1109/VCIP.2017.8305148 ↗
12. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
13. Zhou, L., Zhang, C., and Wu, M. (2018, January 18–22). D-LinkNet: LinkNet with Pretrained Encoder and Dilated Convolution for High Resolution Satellite Imagery Road Extraction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00034 ↗
14. An improved DualGAN for near-infrared image colorization10.1016/j.infrared.2021.103764 ↗
15. Levin, A., Lischinski, D., and Weiss, Y. (2004). Colorization using optimization. ACM SIGGRAPH 2004 Papers, Association for Computing Machinery.10.1145/1186562.1015780 ↗
16. Fast image and video colorization using chrominance blending10.1109/TIP.2005.864231 ↗
17. Zhang, R., Zhu, J.-Y., Isola, P., Geng, X., Lin, A.S., Yu, T., and Efros, A.A. (2017). Real-time user-guided image colorization with learned deep priors. arXiv, preprint.10.1145/3072959.3073703 ↗
18. Welsh, T., Ashikhmin, M., and Mueller, K. (2002, January 23–26). Transferring Color to Greyscale Images. Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, San Antonio, TX, USA.10.1145/566570.566576 ↗
19. Liu, X., Wan, L., Qu, Y., Wong, T.-T., Lin, S., Leung, C.-S., and Heng, P.-A. (2008). Intrinsic colorization. ACM SIGGRAPH Asia 2008 Papers, Association for Computing Machinery.10.1145/1457515.1409105 ↗
20. Colorization by Example
21. Charpiat, G., Hofmann, M., and Schölkopf, B. (2008, January 12–18). Automatic Image Colorization via Multimodal Predictions. Proceedings of the European Conference on Computer Vision, Marseille, France.10.1007/978-3-540-88690-7_10 ↗
22. Gupta, R.K., Chia, A.Y.-S., Rajan, D., Ng, E.S., and Zhiyong, H. (2012, January 3–5). Image Colorization Using Similar Images. Proceedings of the 20th ACM International Conference on Multimedia, Bali, Indonesia.10.1145/2393347.2393402 ↗
23. Semantic colorization with internet images10.1145/2070781.2024190 ↗
24. Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification10.1145/2897824.2925974 ↗
25. Vitoria, P., Raad, L., and Ballester, C. (2020, January 2–5). Chromagan: Adversarial picture colorization with semantic class distribution. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA.10.1109/WACV45572.2020.9093389 ↗
26. Zhang, R., Isola, P., and Efros, A.A. (2016, January 11–14). Colorful Image Colorization. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46487-9_40 ↗
27. Suárez, P.L., Sappa, A.D., and Vintimilla, B.X. (2017, January 6–8). Learning to Colorize Infrared Images. Proceedings of the International Conference on Practical Applications of Agents and Multi-Agent Systems, Salamanca, Spain.10.1007/978-3-319-61578-3_16 ↗
28. Suárez, P.L., Sappa, A.D., and Vintimilla, B.X. (2017, January 21–26). Infrared Image Colorization Based on a Triplet Dcgan Architecture. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.32 ↗
29. Kim, H., Kim, J., and Kim, J. (2021, January 27–30). Infrared Image Colorization Network Using Variational AutoEncoder. Proceedings of the 36th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), IEEE, Jeju-si, Korea.10.1109/ITC-CSCC52171.2021.9605698 ↗
30. Mehri, A., and Sappa, A.D. (2019, January 16–17). Colorizing Near Infrared Images through a Cyclic Adversarial Approach of Unpaired Samples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.10.1109/CVPRW.2019.00128 ↗
31. Zhu, J.-Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
32. Yang, Z., and Chen, Z. (2020, January 1–4). Learning from Paired and Unpaired Data: Alternately Trained CycleGAN for Near Infrared Image Colorization. Proceedings of the 2020 IEEE International Conference on Visual Communications and Image Processing (VCIP), IEEE, Macau, China.10.1109/VCIP49819.2020.9301791 ↗
33. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A Large-Scale Hierarchical Image Database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
34. RGB-NIR Image Enhancement by Fusing Bilateral and Weighted Least Squares Filters10.2352/J.ImagingSci.Technol.2017.61.4.040409 ↗
35. Brown, M., and Süsstrunk, S. (2011, January 20–25). Multi-Spectral SIFT for Scene Category Recognition. Proceedings of the CVPR 2011, IEEE, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995637 ↗
36. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv, preprint.
37. Antic, J. (2022, March 15). A Deep Learning Based Project for Colorizing and Restoring Old Images. Available online: https://github.com/jantic/DeOldifyn.
38. Hore, A., and Ziou, D. (2010, January 23–26). Image Quality Metrics: PSNR vs. SSIM. Proceedings of the 2010 20th International Conference on Pattern Recognition, IEEE, Istanbul, Turkey.10.1109/ICPR.2010.579 ↗
39. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
40. Zhang, X., and Wandell, B.A. (1996, January 12–17). A Spatial Extension of CIELAB for Digital Color Image Reproduction. Proceedings of the SID International Symposium Digest of Technical Papers, San Diego, CA, USA.
41. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–22). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗