An Unsupervised Depth-Estimation Model for Monocular Images Based on Perceptual Image Error Assessment
Source abstract
Abstract
In this paper, we propose a novel unsupervised learning-based model for estimating the depth of monocular images by integrating a simple ResNet-based auto-encoder and some special loss functions. We use only stereo images obtained from binocular cameras as training data without using depth ground-truth data. Our model basically outputs a disparity map that is necessary to warp an input image to an image corresponding to a different viewpoint. When the input image is warped using the output-disparity map, distortions of various patterns inevitably occur in the reconstructed image. During the training process, the occurrence frequency and size of these distortions gradually decrease, while the similarity between the reconstructed and target images increases, which proves that the accuracy of the predicted disparity maps also increases. Therefore, one of the important factors in this type of training is an efficient loss function that accurately measures how much the difference in quality between the reconstructed and target images is and guides the gap to be properly and quickly closed as the training progresses. In recent related studies, the photometric difference was calculated through simple methods such as L1 and L2 loss or by combining one of these with a traditional computer vision-based hand-coded image-quality assessment algorithm such as SSIM. However, these methods have limitations in modeling various patterns at the level of the human visual system. Therefore, the proposed model uses a pre-trained perceptual image-quality assessment model that effectively mimics human-perception mechanisms to measure the quality of distorted images as image-reconstruction loss. In order to highlight the performance of the proposed loss functions, a simple ResNet50-based network is adopted in our model. We trained our model using stereo images of the KITTI 2015 driving dataset to measure the pixel-level depth for 768 × 384 images. Despite the simplicity of the network structure, thanks to the effectiveness of the proposed image-reconstruction loss, our model outperformed other state-of-the-art studies that have been trained in unsupervised methods on a variety of evaluation indicators.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Image and Video Quality Assessment · Image Enhancement Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Software Engineering, Hyupsung University, Hwaseong-si 18330, Korea
Department of Computer Science, Korea University of Technology and Education, Cheonan-si 31253, Korea
Funding
Education and Research Promotion Program of KOREATECH
References (35)
Show source-provided references
1. Godard, C., Aodha, O.M., Firman, M., and Brostow, G.J. (November, January 27). Digging Into Self-Supervised Monocular Depth Estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.
2. Sak, H., Senior, A., Rao, K., and Beaufays, F. (2015). Fast and accurate recurrent neural network acoustic models for speech recognition. arXiv.10.21437/Interspeech.2015-350 ↗
3. Couprie, C., Farabet, C., Najman, L., and LeCun, Y. (2013). Indoor semantic segmentation using depth information. arXiv.
4. Hazirbas, C., Ma, L., Domokos, C., and Cremers, D. (2016). FuseNet: Incorporating depth into semantic segmentation via fusion-based CNN architecture. Computer Vision–ACCV 2016, Springer. Lecture Notes in Computer Science.10.1007/978-3-319-54181-5_14 ↗
5. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
6. Godard, C., Aodha, O.M., and Brostow, G.J. (2017, January 21–26). Unsupervised Monocular Depth Estimation With left–right Consistency. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.699 ↗
7. Relativistic Approach for Training Self-Supervised Adversarial Depth Prediction Model Using Symmetric Consistency10.1109/ACCESS.2020.3036893 ↗
8. Image Quality Assessment: Unifying Structure and Texture Similarity
9. Stereo matching by training a convolutional neural network to compare image patches
10. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗
11. Prashnani, E., Cai, H., Mostofi, Y., and Sen, P. (2018, January 18–23). PieAPP: Perceptual Image-Error Assessment Through Pairwise Preference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00194 ↗
12. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
13. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
14. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The Cityscapes Dataset for Semantic Urban Scene Understanding. Proceedings of the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
15. Eigen, D., Puhrsch, C., and Fergus, R. (2014, January 8–13). Depth map prediction from a single image using a multi-scale deep network. Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
16. Liu, M., Salzmann, M., and He, X. (2014, January 23–28). Discrete-Continuous Depth Estimation from a Single Image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.97 ↗
17. Make3D: Learning 3D Scene Structure from a Single Still Image10.1109/TPAMI.2008.132 ↗
18. Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. (2012). Indoor segmentation and support inference from rgbd images. European Conference on Computer Vision, Springer.10.1007/978-3-642-33715-4_54 ↗
19. Fu, H., Gong, M., Wang, C., Batmanghelich, K., and Tao, D. (2018, January 18–23). Deep Ordinal Regression Network for Monocular Depth Estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00214 ↗
20. Zhou, T., Brown, M., Snavely, N., and Lowe, D.G. (2017, January 21–26). Unsupervised Learning of Depth and Ego-Motion From Video. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.700 ↗
21. Mayer, N., Ilg, E., Häusser, P., Fischer, P., Cremers, D., Dosovitskiy, A., and Brox, T. (2016, January 27–30). A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.438 ↗
22. Yang, Z., Wang, P., Xu, W., Zhao, L., and Nevatia, R. (2018). Unsupervised learning of geometry with edge-aware depth-normal consistency. arXiv.10.1609/aaai.v32i1.12257 ↗
23. Mahjourian, R., Wicke, M., and Angelova, A. (2018, January 18–23). Unsupervised Learning of Depth and Ego-Motion From Monocular Video Using 3D Geometric Constraints. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00594 ↗
24. Yin, Z., and Shi, J. (2018, January 18–23). GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00212 ↗
25. Wang, C., Buenaposada, J.M., Zhu, R., and Lucey, S. (2018, January 18–23). Learning Depth From Monocular Videos Using Direct Methods. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00216 ↗
26. Engel, J., Schöps, T., and Cremers, D. (2014). LSD-SLAM: Large-scale direct monocular SLAM. European Conference on Computer Vision, Springer.10.1007/978-3-319-10605-2_54 ↗
27. Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding10.1109/TPAMI.2019.2930258 ↗
28. Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
29. Pillai, S., Ambruş, R., and Gaidon, A. (2019, January 20–24). SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.10.1109/ICRA.2019.8793621 ↗
30. Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index10.1109/TIP.2013.2293423 ↗
31. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative Adversarial Nets. Proceedings of the Advances in Neural Information Processing Systems 27 (NIPS 2014), Montreal, QC, Canada.
32. Jolicoeur-Martineau, A. (2019, January 6–9). The relativistic discriminator: A key element missing from standard GAN. Proceedings of the International Conference on Learning Representations (ICLR) 2019, New Orleans, LA, USA.
33. Kastryulin, S., Zakirov, D., and Prokopenko, D. (2022, March 03). PyTorch Image Quality: Metrics and Measure for Image Quality Assessment. Available online: https://github.com/photosynthesis-team/piq/.10.2139/ssrn.4206741 ↗
34. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 4–9). Automatic differentiation in PyTorch. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
35. Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv.