journal article·2023-01-29Open access

Retinex-Based Relighting for Night Photography

Sou Oishi, Norishige Fukushima

Applied Sciences· Volume 13· Issue 3

Abstract

The lighting up of buildings is one form of entertainment that makes a city more colorful, and photographers sometimes change this lighting using photo-editing applications. This paper proposes a method for automatically performing such changes that follows the Retinex theory. Retinex theory indicates that the complex scenes caught by the human visual system are affected by surrounding colors, and Retinex-based image processing uses these characteristics to generate images. Our proposed method follows this approach. First, we propose a method for extracting a relighting saliency map using Retinex with edge-preserving filtering. Second, we propose a sampling method to specify the lighting area. Finally, we composite the additional light to match the human visual perception. Experimental results show that the proposed sampling method is successful in keeping the illuminated points in bright locations and equally spaced apart. In addition, the proposed various diffusion methods can enhance nighttime skyline photographs with various expressions. Finally, we can add in a new light by considering Retinex theory to represent the perceptual color.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Image Enhancement Techniques · Visual Attention and Saliency Detection · Color Science and Applications

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Nagoya Institute of Technology, Nagoya 464-8602, Japan

Funding

JSPS KAKENHI

References (62)

Show source-provided references
1. The retinex
2. The retinex theory of color vision10.1038/scientificamerican1277-108 ↗
3. An alternative technique for the computation of the designator in the retinex theory of color vision10.1073/pnas.83.10.3078 ↗
4. Retinex processing for automatic image enhancement10.1117/1.1636183 ↗
5. LIME: Low-light image enhancement via illumination map estimation10.1109/TIP.2016.2639450 ↗
6. LR3M: Robust low-light enhancement via low-rank regularized retinex model10.1109/TIP.2020.2984098 ↗
7. Star: A structure and texture aware retinex model10.1109/TIP.2020.2974060 ↗
8. Low-light image enhancement via poisson noise aware retinex model10.1109/LSP.2021.3096160 ↗
9. Tang, H., Zhu, H., Tao, H., and Xie, C. (2022). An Improved Algorithm for Low-Light Image Enhancement Based on RetinexNet. Appl. Sci., 12.10.3390/app12147268 ↗
10. Pan, X., Li, C., Pan, Z., Yan, J., Tang, S., and Yin, X. (2022). Low-Light Image Enhancement Method Based on Retinex Theory by Improving Illumination Map. Appl. Sci., 12.10.3390/app12105257 ↗
11. High dynamic range image rendering with a retinex-based adaptive filter10.1109/TIP.2006.877312 ↗
12. Parthasarathy, S., and Sankaran, P. (2012, January 6–9). A RETINEX based haze removal method. Proceedings of the IEEE International Conference on Industrial and Information Systems (ICIIS), Chennai, India.10.1109/ICIInfS.2012.6304767 ↗
13. Galdran, A., Alvarez-Gila, A., Bria, A., Vazquez-Corral, J., and Bertalmío, M. (2018, January 18–23). On the duality between retinex and image dehazing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00857 ↗
14. Bayesian retinex underwater image enhancement10.1016/j.engappai.2021.104171 ↗
15. Finlayson, G.D., Hordley, S.D., and Drew, M.S. (2002, January 12–15). Removing shadows from images using retinex. Proceedings of the Color and Imaging Conference, Scottsdale, AZ, USA.10.2352/CIC.2002.10.1.art00016 ↗
16. From retinex to automatic color equalization: Issues in developing a new algorithm for unsupervised color equalization10.1117/1.1635366 ↗
17. Okuhata, H., Nakamura, H., Hara, S., Tsutsui, H., and Onoye, T. (2013, January 3–7). Application of the real-time Retinex image enhancement for endoscopic images. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan.10.1109/EMBC.2013.6610273 ↗
18. Almalki, Y.E., Jandan, N.A., Soomro, T.A., Ali, A., Kumar, P., Irfan, M., Keerio, M.U., Rahman, S., Alqahtani, A., and Alqhtani, S.M. (2022). Enhancement of Medical Images through an Iterative McCann Retinex Algorithm: A Case of Detecting Brain Tumor and Retinal Vessel Segmentation. Appl. Sci., 12.10.3390/app12168243 ↗
19. Retinex method based on adaptive smoothing for illumination invariant face recognition10.1016/j.sigpro.2008.01.028 ↗
20. Low-light image enhancement using variational optimization-based retinex model10.1109/TCE.2017.014847 ↗
21. Properties and performance of a center/surround retinex10.1109/83.557356 ↗
22. Eisemann, E., and Durand, F. (2004). Flash Photography Enhancement via Intrinsic Relighting. ACM Trans. Graph., 673–678.10.1145/1015706.1015778 ↗
23. Digital Photography with Flash and No-flash Image Pairs10.1145/1015706.1015777 ↗
24. Recursive erosion, dilation, opening, and closing transforms10.1109/83.366481 ↗
25. Otsuka, T., Fukushima, N., Maeda, Y., Sugimoto, K., and Kamata, S. (2020, January 1–4). Optimization of Sliding-DCT based Gaussian Filtering for Hardware Accelerator. Proceedings of the International Conference on Visual Communications and Image Processing (VCIP), Macau, China.10.1109/VCIP49819.2020.9301775 ↗
26. Sumiya, Y., Fukushima, N., Sugimoto, K., and Kamata, S. (2020, January 25–28). Extending Compressive Bilateral Filtering for Arbitrary Range Kernel. Proceedings of the IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab.10.1109/ICIP40778.2020.9191123 ↗
27. Single image haze removal using dark channel prior
28. He, K., Shun, J., and Tang, X. (2010, January 5–11). Guided Image Filtering. Proceedings of the European Conference on Computer Vision (ECCV), Heraklion, Crete, Greece.10.1007/978-3-642-15549-9_1 ↗
29. Fukushima, N., Sugimoto, K., and Kamata, S. (2018, January 15–20). Guided Image Filtering with Arbitrary Window Function. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada.10.1109/ICASSP.2018.8462016 ↗
30. Stochastic sampling in computer graphics10.1145/7529.8927 ↗
31. Non-Local Retinex—A Unifying Framework and Beyond10.1137/140972664 ↗
32. Retinex at 50: Color theory and spatial algorithms, a review10.1117/1.JEI.26.3.031204 ↗
33. Determining lightness from an image10.1016/0146-664X(74)90022-7 ↗
34. A PDE formalization of Retinex theory10.1109/TIP.2010.2049239 ↗
35. Retinex in MATLABTM10.1117/1.1636761 ↗
36. Rahman, Z.u., Jobson, D.J., and Woodell, G.A. (1996, January 19). Multi-scale retinex for color image enhancement. Proceedings of the IEEE international conference on image processing (ICIP), Lausanne, Switzerland.
37. A multiscale retinex for bridging the gap between color images and the human observation of scenes10.1109/83.597272 ↗
38. A variational framework for retinex10.1023/A:1022314423998 ↗
39. Elad, M. (2005, January 7–9). Retinex by two bilateral filters. Proceedings of the International Conference on Scale-Space Theories in Computer Vision, Hofgeismar, Germany.10.1007/11408031_19 ↗
40. Wei, C., Wang, W., Yang, W., and Liu, J. (2018, January 3–6). Deep Retinex Decomposition for Low-Light Enhancement. Proceedings of the British Machine Vision Conference (BMVC), Newcastle, UK.
41. Deep retinex network for single image dehazing10.1109/TIP.2020.3040075 ↗
42. Grosse, R., Johnson, M., Adelson, E., and Freeman, W. (October, January 29). Ground-truth dataset and baseline evaluations for intrinsic image algorithms. Proceedings of the International Conference on Computer Vision (ICCV), Kyoto, Japan.
43. Gehler, P., Rother, C., Kiefel, M., Zhang, L., and Scholkopf, B. (2011, January 12–17). Recovering intrinsic images with a global sparsity prior on reflectance. Proceedings of the Neural Information Processing Systems (NIPS), Granada, Spain.
44. Barron, J., and Malik, J. (2012, January 7–13). Color constancy, intrinsic images, and shape estimation. Proceedings of the European Conference on Computer Vision (ECCV), Firenze, Italy.10.1007/978-3-642-33765-9_5 ↗
45. Li, Y., and Brown, M. (2014, January 23–28). Single image layer separation using relative smoothness. Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.346 ↗
46. Lightness and retinex theory10.1364/JOSA.61.000001 ↗
47. Tuning retinex parameters10.1117/1.1635365 ↗
48. A model of saliency-based visual attention for rapid scene analysis10.1109/34.730558 ↗
49. Efficient constant-time Gaussian filtering with sliding DCT/DST-5 and dual-domain error minimization10.3169/mta.3.12 ↗
50. Sugimoto, K., Kyochi, S., and Kamata, S. (2018, January 15–20). Universal approach for DCT-based constant-time Gaussian filter with moment preservation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461679 ↗
51. Fukushima, N., Maeda, Y., Kawasaki, Y., Nakamura, M., Tsumura, T., Sugimoto, K., and Kamata, S. (2018, January 12–15). Efficient Computational Scheduling of Box and Gaussian FIR Filtering for CPU Microarchitecture. Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Honolulu, HI, USA.10.23919/APSIPA.2018.8659674 ↗
52. Tomasi, C., and Manduchi, R. (1998, January 7). Bilateral Filtering for Gray and Color Images. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Bombay, India.
53. Sugimoto, K., Fukushima, N., and Kamata, S. (2019, January 22–25). 200 FPS Constant-time Bilateral Filter Using SVD and Tiling Strategy. Proceedings of the IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8802927 ↗
54. Dithering with blue noise10.1109/5.3288 ↗
55. Mitsa, T., and Parker, K. (1991, January 14–17). Digital halftoning using a blue noise mask. Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP.1991.150986 ↗
56. Image Perforation: Automatically Accelerating Image Pipelines by Intelligently Skipping Samples10.1145/2904903 ↗
57. Domain Transform for Edge-Aware Image and Video Processing10.1145/2010324.1964964 ↗
58. Adaptive Manifolds for Real-Time High-Dimensional Filtering10.1145/2185520.2185529 ↗
59. Wu, W., Weng, J., Zhang, P., Wang, X., Yang, W., and Jiang, J. (2022, January 18–24). URetinex-Net: Retinex-Based Deep Unfolding Network for Low-Light Image Enhancement. Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00581 ↗
60. Gayoung, L., Yu-Wing, T., and Junmo, K. (2016, January 27–30). Deep Saliency with Encoded Low level Distance Map and High Level Features. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
61. Maeda, Y., Fukushima, N., and Matsuo, H. (2018). Effective Implementation of Edge-Preserving Filtering on CPU Microarchitectures. Appl. Sci., 8.10.3390/app8101985 ↗
62. Maeda, Y., Fukushima, N., and Matsuo, H. (2018). Taxonomy of Vectorization Patterns of Programming for FIR Image Filters Using Kernel Subsampling and New One. Appl. Sci., 8.10.3390/app8081235 ↗