Extending Light Direction Reconstruction to Outdoor and Surface Datasets
Source abstract
Abstract
Immersive augmented reality (AR) requires consistent illumination between synthetic scene content and the real world. Recent deep neural networks achieve this by reconstructing important illumination parameters, though the performance was measured in clean laboratory environments. This study shows that the reconstruction performance of a recently proposed network architecture remains satisfying in more complex real-world outdoor scenarios. The labelled outdoor scenario dataset used in this study is provided as part of this publication. The study further reveals how auxiliary surface (in this scope, depth and normal vector) information affects the reconstruction performance of the network architecture.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Computer Graphics and Visualization Techniques · Advanced Vision and Imaging · Augmented Reality Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Mathematics, University of Applied Sciences Munich, Lothstr. 64, D-80335 Munich, Bavaria, Germany
Chair of Computer Graphics and Visualization, Technical University of Munich, Boltzmannstr. 3/II, D-85748 Garching, Bavaria, Germany
References (26)
Show source-provided references
1. Kanbara, M., and Yokoya, N. (2002, January 1). Geometric and photometric registration for real-time augmented reality. Proceedings of the International Symposium on Mixed and Augmented Reality, Darmstadt, Germany.
2. DeepLight: Light Source Estimation for Augmented Reality using Deep Learning10.1007/s00371-019-01666-x ↗
3. Miller, M., Nischwitz, A., and Westermann, R. (2021, January 17–21). Deep Light Direction Reconstruction from single RGB images. Proceedings of the 29 International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision: Full Papers Proceedings, Plzen, Czech Republic. Computer Science Research Notes.10.24132/CSRN.2021.3101.4 ↗
4. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very deep convolutional networks for large-scale image recognition. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015: Conference Track Proceedings, San Diego, CA, USA.
5. Light Direction Reconstruction Analysis and Improvement using XAI and CG
6. The Rendering Equation10.1145/15886.15902 ↗
7. Deep multimodal learning: A survey on recent advances and trends10.1109/MSP.2017.2738401 ↗
8. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
9. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A ConvNet for the 2020s. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
10. Learning to Predict Indoor Illumination from a Single Image10.1145/3130800.3130891 ↗
11. Garon, M., Sunkavalli, K., Hadap, S., Carr, N., and Lalonde, J. (2019, January 15–20). Fast Spatially-Varying Indoor Lighting Estimation. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00707 ↗
12. Gardner, M.A., Hold-Geoffroy, Y., Sunkavalli, K., Gagné, C., and Lalonde, J.F. (November, January 27). Deep Parametric Indoor Lighting Estimation. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00727 ↗
13. Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., and Lalonde, J. (2017, January 21–26). Deep outdoor illumination estimation. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.255 ↗
14. Hold-Geoffroy, Y., Athawale, A., and Lalonde, J.F. (2019, January 15–20). Deep Sky Modeling for Single Image Outdoor Lighting Estimation. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00709 ↗
15. LeGendre, C., Ma, W.C., Fyffe, G., Flynn, J., Charbonnel, L., Busch, J., and Debevec, P. (August, January 28). DeepLight: Learning illumination for unconstrained mobile mixed reality. Proceedings of the ACM SIGGRAPH 2019, Talks, Los Angeles, CA, USA. SIGGRAPH 2019.10.1145/3306307.3328173 ↗
16. Real-time Light Estimation and Neural Soft Shadows for AR Indoor Scenarios10.24132/JWSCG.2023.8 ↗
17. Oren, M., and Nayar, S.K. (1994, January 19–23). Generalization of Lambert’s Reflectance Model. Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA. SIGGRAPH ’94.10.1145/192161.192213 ↗
18. Cook, R.L., and Torrance, K.E. (1981, January 3–7). A Reflectance Model for Computer Graphics. Proceedings of the 8th Annual Conference on Computer Graphics and Interactive Techniques, Dallas, TX, USA. SIGGRAPH ’81.10.1145/800224.806819 ↗
19. Boksansky, J., Wimmer, M., and Bittner, J. (2019). Ray Traced Shadows: Maintaining Real-Time Frame Rates. Ray Tracing Gems: High-Quality and Real-Time Rendering with DXR and Other APIs, Apress.10.1007/978-1-4842-4427-2_13 ↗
20. (2024, March 01). Keras Tuner, Version 1.4.7. Available online: https://github.com/keras-team/keras-tuner.
21. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
22. Nakagawa, Y., Uchiyama, H., Nagahara, H., and Taniguchi, R.I. (2015, January 19–22). Estimating Surface Normals with Depth Image Gradients for Fast and Accurate Registration. Proceedings of the 2015 International Conference on 3D Vision, Lyon, France.10.1109/3DV.2015.80 ↗
23. Improved Kiwifruit Detection Using Pre-Trained VGG16 with RGB and NIR Information Fusion10.1109/ACCESS.2019.2962513 ↗
24. Individual Comparisons by Ranking Methods10.2307/3001968 ↗
25. Notes on regression and inheritance in the case of two parents10.1098/rspl.1895.0041 ↗
26. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, Lawrence Erlbaum Associates. [2nd ed.].