Monocular Depth Estimation Using Res-UNet with an Attention Model
Source abstract
Abstract
Depth maps are single image metrics that carry the information of a scene in three-dimensional axes. Accurate depth maps can recreate the 3D structure of a scene, which helps in understanding the full geometry of the objects within the scene. Depth maps can be generated from a single image or multiple images. Single-image depth mapping is also known as monocular depth mapping. Depth maps are ill-posed problems that are complex and require extensive calibration. Therefore, recent methods use deep learning to develop depth maps. We propose a new method in monocular depth estimation to develop a high-quality depth map. Our approach is based on a convolutional neural network in which we used Res-UNet with a spatial attention model to develop depth maps. The addition of an attention mechanism increases the capability of feature extraction and enhances the boundaries features. It does not add any extra parameters to the network. With our proposed model, we demonstrate that a simple CNN model aided with an attention mechanism can create high-quality depth maps with a small iteration and training time. Our model performs very well compared to the existing state-of-the-art methods on the benchmark NYU-depth v2 dataset. Our model is flexible and can be applied to any depth mapping or multi-segmentation tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Image Processing Techniques and Applications · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Civil Engineering, School of Architectural, Civil, Environmental, and Energy Engineering, Kyungpook National University, Daegu 41566, Republic of Korea
Funding
Ministry of Education
References (54)
Show source-provided references
1. Hartley, R., and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, Cambridge University Press. [2nd ed.].10.1017/CBO9780511811685 ↗
2. Mobile Robot Localization by Tracking Geometric Beacons10.1109/70.88147 ↗
3. Leonard, J.J., Durrant-Whyte, H.F., and Pj, O. (2002). Directed Sonar Sensing for Mobile Robot Navigation, Springer Science & Business Media.
4. Indoor Segmentation and Support Inference from RGBD Images10.1007/978-3-642-33715-4_54 ↗
5. Eigen, D., Puhrsch, C., and Fergus, R. (2014). Depth Map Prediction from a Single Image Using a Multi-Scale Deep Network. arXiv.
6. Eigen, D., and Fergus, R. (2015). Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture. arXiv.10.1109/ICCV.2015.304 ↗
7. Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields10.1109/TPAMI.2015.2505283 ↗
8. Deep Learning-Based Depth Map Estimation: A Review
9. Alhashim, I., and Wonka, P. (2018). High Quality Monocular Depth Estimation via Transfer Learning. arXiv.
10. Garg, R., Bg, V.K., Carneiro, G., and Reid, I. (2016). Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VIII 14 2016, Springer International Publishing.
11. Godard, C., Mac Aodha, O., and Brostow, G.J. (2017). Unsupervised Monocular Depth Estimation with Left-Right Consistency. arXiv.10.1109/CVPR.2017.699 ↗
12. Vasiljevic, I., Kolkin, N., Zhang, S., Luo, R., Wang, H., Dai, F.Z., Daniele, A.F., Mostajabi, M., Basart, S., and Walter, M.R. (2019). DIODE: A Dense Indoor and Outdoor DEpth Dataset. arXiv.
13. XVIII. Contributions to the Physiology of Vision—Part the First. On Some Remarkable, and Hitherto Unobserved, Phenomena of Binocular Vision10.1098/rstl.1838.0019 ↗
14. Helmholtz’s Treatise on Physiological Optics10.1038/116088a0 ↗
15. Automatic Photo Pop-up: ACM SIGGRAPH 2005
16. Depth Transfer: Depth Extraction from Video Using Non-Parametric Sampling10.1109/TPAMI.2014.2316835 ↗
17. Ladický, L., Shi, J., and Pollefeys, M. (2014, January 23–28). Pulling Things out of Perspective. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.19 ↗
18. Saxena, A., Chung, S., and Ng, A. (2005, January 5–8). Learning Depth from Single Monocular Images. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
19. Yang, J., Alvarez, J.M., and Liu, M. (2022, January 18–24). Non-Parametric Depth Distribution Modelling Based Depth Inference for Multi-View Stereo. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00843 ↗
20. Introducing the Discrete Path Transform (DPT) and Its Applications in Signal Analysis, Artefact Removal, and Spoken Word Recognition10.1016/j.dsp.2021.103158 ↗
21. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
22. Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv.
23. Seo, S. (2021). SNR Analysis for Quantitative Comparison of Line Detection Methods. Appl. Sci., 11.10.3390/app112110088 ↗
24. Hyperspectral Image Classification Using Wavelet Transform-Based Smooth Ordering10.1142/S0219691319500504 ↗
25. A Framework of Adaptive Multiscale Wavelet Decomposition for Signals on Undirected Graphs10.1109/TSP.2019.2896246 ↗
26. Silvestrov, S., and Rančić, M. (2016). Engineering Mathematics II: Algebraic, Stochastic and Analysis Structures for Networks, Data Classification and Optimization, Springer International Publishing.10.1007/978-3-319-42105-6 ↗
27. Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., and Navab, N. (2016). Deeper Depth Prediction with Fully Convolutional Residual Networks. arXiv.10.1109/3DV.2016.32 ↗
28. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
29. Zwald, L., and Lambert-Lacroix, S. (2012). The BerHu Penalty and the Grouped Effect. arXiv.
30. Wang, P., Cohen, S., Price, B., and Yuille, A. (2015, January 7–12). Towards Unified Depth and Semantic Prediction from a Single Image. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
31. Roy, A., and Todorovic, S. (2016, January 27–30). Monocular Depth Estimation Using Neural Regression Forest. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.594 ↗
32. Fu, H., Gong, M., Wang, C., Batmanghelich, K., and Tao, D. (2018, January 18–23). Deep Ordinal Regression Network for Monocular Depth Estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00214 ↗
33. Hu, J., Ozay, M., Zhang, Y., and Okatani, T. (2018). Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries. arXiv.10.1109/WACV.2019.00116 ↗
34. Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K.Q. (2016). Densely Connected Convolutional Networks. arXiv.10.1109/CVPR.2017.243 ↗
35. Kumari, S., Jha, R.R., Bhavsar, A., and Nigam, A. (2019, January 22–25). AUTODEPTH: Single Image Depth Map Estimation via Residual CNN Encoder-Decoder and Stacked Hourglass. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803006 ↗
36. Zhou, T., Brown, M., Snavely, N., and Lowe, D.G. (2017). Unsupervised Learning of Depth and Ego-Motion from Video. arXiv.10.1109/CVPR.2017.700 ↗
37. Casser, V., Pirk, S., Mahjourian, R., and Angelova, A. (2018). Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos. arXiv.10.1609/aaai.v33i01.33018001 ↗
38. Chen, Z., Badrinarayanan, V., Drozdov, G., and Rabinovich, A. (2018). Estimating Depth from RGB and Sparse Sensing. arXiv.10.1007/978-3-030-01225-0_11 ↗
39. Mahjourian, R., Wicke, M., and Angelova, A. (2018). Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints. arXiv.10.1109/CVPR.2018.00594 ↗
40. Goldman, M., Hassner, T., and Avidan, S. (2019). Learn Stereo, Infer Mono: Siamese Networks for Self-Supervised, Monocular, Depth Estimation. arXiv.10.1109/CVPRW.2019.00348 ↗
41. Koch, G., Zemel, R., and Salakhutdinov, R. (2023, May 10). Siamese Neural Networks for One-Shot Image Recognition. Available online: http://www.cs.toronto.edu/~gkoch/files/msc-thesis.pdf.
42. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–22). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
43. Song, J., Kim, S., and Yoon, S. (2021). AligNART: Non-Autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate. arXiv.10.18653/v1/2021.emnlp-main.1 ↗
44. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. 2017. arXiv.
45. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2023, May 10). Learning Deep Features for Discriminative Localization 2015. Available online: https://arxiv.org/abs/1512.04150.
46. Mnih, V., Heess, N., and Graves, A. (2014, January 8–13). Kavukcuoglu, Koray Recurrent Models of Visual Attention. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
47. Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., and Tang, X. (2017, January 21–26). Residual Attention Network for Image Classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.683 ↗
48. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
49. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., and Bengio, Y. (2015, January 6–11). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Proceedings of the 32nd International Conference on Machine Learning, Lille, France.
50. Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., and Kainz, B. (2018). Attention U-Net: Learning Where to Look for the Pancreas. arXiv.
51. Ronneberger, O., Fischer, P., and Brox, T. (2015). Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18, Springer International Publishing.
52. Huang, P.-H., Matzen, K., Kopf, J., Ahuja, N., and Huang, J.-B. (2018, January 18–23). DeepMVS: Learning Multi-View Stereopsis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00298 ↗
53. Ummenhofer, B., Zhou, H., Uhrig, J., Mayer, N., Ilg, E., Dosovitskiy, A., and Brox, T. (2017). DeMoN: Depth and Motion Network for Learning Monocular Stereo. arXiv.10.1109/CVPR.2017.596 ↗
54. Kingma, D.P., and Ba, J. (2017). Adam: A Method for Stochastic Optimization. arXiv.