Attention-Based Background/Foreground Monocular Depth Prediction Model Using Image Segmentation
Source abstract
Abstract
While many monocular depth estimation methods have been proposed, determining depth variations in outdoor scenes remains challenging. Accordingly, this paper proposes an image segmentation-based monocular depth estimation model with attention mechanisms that can address outdoor scene variations. The segmentation model segments images into foreground and background regions and individually predicts depth maps. Moreover, attention mechanisms are also adopted to extract meaningful features from complex scenes to improve foreground and background depth map prediction via a multi-scale decoding scheme. From our experimental results, we observed that our proposed model outperformed previous methods by 27.5% on the KITTI dataset.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Image Processing Techniques and Applications · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Engineering and Computer Science, Feng Chia University, Taichung 407102, Taiwan
Funding
National Science and Technology Council (NSTC)
References (69)
Show source-provided references
1. Liu, Y., Jiang, J., Sun, J., Bai, L., and Wang, Q. (2020, January 27–29). A Survey of Depth Estimation Based on Computer Vision. Proceedings of the IEEE 5th International Conference on Data Science Cyberspace, Hong Kong, China.10.1109/DSC50466.2020.00028 ↗
2. Eigen, D., Puhrsch, C., and Fergus, R. (2014, January 8–13). Depth Map Prediction from a Single Image using a Multi-Scale Deep Network. Proceedings of the Neural Information Processing Systems, Montreal, QC, USA.
3. Eigen, D., and Fergus, R. (2015, January 7–13). Predicting Depth, Surface Normals and Semantic Labels With a Common Multi-Scale Convolutional Architecture. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.304 ↗
4. Depth estimation with convolutional conditional random field network10.1016/j.neucom.2016.06.029 ↗
5. Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., and Navab, N. (2016, January 25–28). Deeper Depth Prediction with Fully Convolutional Residual Networks. Proceedings of the 4th International Conference on 3D Vision, Stanford, CA, USA.10.1109/3DV.2016.32 ↗
6. Hu, J., Ozay, M., Zhang, Y., and Okatani, T. (2019, January 7–11). Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps With Accurate Object Boundaries. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Waikoloa Village, HI, USA.10.1109/WACV.2019.00116 ↗
7. Fu, H., Gong, M., Wang, C., Batmanghelich, K., and Tao, D. (2018, January 18–23). Deep Ordinal Regression Network for Monocular Depth Estimation. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00214 ↗
8. Li, B., Shen, C., Dai, Y., van den Hengel, A., and He, M. (2015, January 7–12). Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Boston, MA, USA.
9. Mousavian, A., Pirsiavash, H., and Košecká, J. (2016, January 25–28). Joint Semantic Segmentation and Depth Estimation with Deep Convolutional Networks. Proceedings of the 4th International Conference on 3D Vision, Stanford, CA, USA.10.1109/3DV.2016.69 ↗
10. Lee, J.H., Heo, M., Kim, K.R., and Kim, C.S. (2018, January 18–23). Single-Image Depth Estimation Based on Fourier Domain Analysis. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00042 ↗
11. Cheng, X., Wang, P., and Yang, R. (2018, January 8–14). Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01270-0_7 ↗
12. Lee, J.H., Han, M.K., Ko, D.W., and Suh, I.H. (2019). From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv.
13. Depth Map Enhancement by Revisiting Multi-Scale Intensity Guidance Within Coarse-to-Fine Stages10.1109/TCSVT.2019.2962867 ↗
14. Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals10.1109/TCSVT.2021.3049869 ↗
15. Bian, J., Li, Z., Wang, N., Zhan, H., Shen, C., Cheng, M.M., and Reid, I. (2019, January 8–14). Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video. Proceedings of the Neural Information Processing Systems, Vancouver, BC, Canada.
16. Godard, C., Mac Aodha, O., and Brostow, G.J. (2017, January 21–26). Unsupervised Monocular Depth Estimation With Left-Right Consistency. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.699 ↗
17. Garg, R., Bg, V.K., Carneiro, G., and Reid, I. (2016, January 11–14). Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_45 ↗
18. Luo, Y., Ren, J., Lin, M., Pang, J., Sun, W., Li, H., and Lin, L. (2018, January 18–23). Single View Stereo Matching. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00024 ↗
19. Watson, J., Firman, M., Brostow, G.J., and Turmukhambetov, D. (2019, January 27–28). Self-Supervised Monocular Depth Hints. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00225 ↗
20. Depth Prediction without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos
21. Poggi, M., Tosi, F., and Mattoccia, S. (2018, January 5–8). Learning Monocular Depth Estimation with Unsupervised Trinocular Assumptions. Proceedings of the 6th International Conference on 3D Vision, Verona, Italy.10.1109/3DV.2018.00045 ↗
22. Zhou, T., Brown, M., Snavely, N., and Lowe, D.G. (2017, January 21–26). Unsupervised Learning of Depth and Ego-Motion from Video. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.700 ↗
23. Godard, C., Mac Aodha, O., Firman, M., and Brostow, G.J. (2019, January 27–28). Digging Into Self-Supervised Monocular Depth Estimation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00393 ↗
24. Guizilini, V., Ambrus, R., Pillai, S., Raventos, A., and Gaidon, A. (2020, January 14–19). 3D Packing for Self-Supervised Monocular Depth Estimation. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00256 ↗
25. Chen, L., Yang, Z., Ma, J., and Luo, Z. (2018, January 12–15). Driving Scene Perception Network: Real-Time Joint Detection, Depth Estimation and Semantic Segmentation. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, NV, USA.10.1109/WACV.2018.00145 ↗
26. Nekrasov, V., Dharmasiri, T., Spek, A., Drummond, T., Shen, C., and Reid, I. (2019, January 20–24). Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations. Proceedings of the International Conference on Robotics and Automation, Montreal, QC, Canada.10.1109/ICRA.2019.8794220 ↗
27. Chen, P.Y., Liu, A.H., Liu, Y.C., and Wang, Y.C.F. (2019, January 15–20). Towards Scene Understanding: Unsupervised Monocular Depth Estimation With Semantic-Aware Representation. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00273 ↗
28. Wang, P., Shen, X., Lin, Z., Cohen, S., Price, B., and Yuille, A. (2015, January 7–12). Towards unified depth and semantic prediction from a single image. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Boston, MA, USA.
29. Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., and Yang, J. (2019, January 15–20). Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00423 ↗
30. SOSD-Net: Joint semantic object segmentation and depth estimation from monocular images10.1016/j.neucom.2021.01.126 ↗
31. Zhu, S., Brazil, G., and Liu, X. (2020, January 13–19). The Edge of Depth: Explicit Constraints Between Segmentation and Depth. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01313 ↗
32. Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation10.1109/TNNLS.2017.2787781 ↗
33. Zhang, Z., Cui, Z., Xu, C., Jie, Z., Li, X., and Yang, J. (2018, January 8–14). Joint Task-Recursive Learning for Semantic Segmentation and Depth Estimation. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01249-6_15 ↗
34. Xu, D., Ricci, E., Ouyang, W., Wang, X., and Sebe, N. (2017, January 21–26). Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.25 ↗
35. Xu, D., Wang, W., Tang, H., Liu, H., Sebe, N., and Ricci, E. (2018, January 18–23). Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00412 ↗
36. Lee, S., Lee, J., Kim, B., Yi, E., and Kim, J. (2021, January 2–9). Patch-wise attention network for monocular depth estimation. Proceedings of the AAAI Conference on Artificial Intelligence, Online.10.1609/aaai.v35i3.16282 ↗
37. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
38. Make3D: Learning 3D Scene Structure from a Single Still Image10.1109/TPAMI.2008.132 ↗
39. Monocular depth estimation based on deep learning: An overview10.1007/s11431-020-1582-8 ↗
40. Yin, W., Liu, Y., Shen, C., and Yan, Y. (2019, January 15–20). Enforcing Geometric Constraints of Virtual Normal for Depth Prediction. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Long Beach, CA, USA.10.1109/ICCV.2019.00578 ↗
41. Kuznietsov, Y., Stückler, J., and Leibe, B. (2017, January 21–26). Semi-Supervised Deep Learning for Monocular Depth Map Prediction. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.238 ↗
42. Guo, M., Xu, T., Liu, J., Liu, Z., Jiang, P., Mu, T., Zhang, S., Martin, R.R., Cheng, M., and Hu, S. (2021). Attention mechanisms in computer vision: A survey. arXiv.
43. Liu, S., Johns, E., and Davison, A.J. (2019, January 15–20). End-To-End Multi-Task Learning With Attention. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00197 ↗
44. Squeeze-and-Excitation Networks10.1109/TPAMI.2019.2913372 ↗
45. Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8–14). Cbam: Convolutional block attention module. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
46. Yan, J., Zhao, H., Bu, P., and Jin, Y. (2021, January 12–15). Channel-Wise Attention-Based Network for Self-Supervised Monocular Depth Estimation. Proceedings of the International Conference on 3D Vision, Prague, Czech Republic.10.1109/3DV53792.2021.00056 ↗
47. Yang, G., Rota, P., Alameda-Pineda, X., Xu, D., Ding, M., and Ricci, E. (2022). Variational Structured Attention Networks for Deep Visual Representation Learning. IEEE Trans. Image Process., Early Access.
48. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Las Vegas, NV, USA.
49. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
50. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., and Hu, Q. (2020, January 14–19). ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01155 ↗
51. Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., and Agrawal, A. (2018, January 18–23). Context Encoding for Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00747 ↗
52. Li, X., Wang, W., Hu, X., and Yang, J. (2019, January 15–20). Selective Kernel Networks. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00060 ↗
53. Gao, Z., Xie, J., Wang, Q., and Li, P. (2019, January 15–20). Global Second-Order Pooling Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00314 ↗
54. Lee, H., Kim, H.E., and Nam, H. (November, January 27). SRM: A Style-Based Recalibration Module for Convolutional Neural Networks. Proceedings of the International Conference on Computer Vision, Seoul, Korea.
55. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-End Object Detection with Transformers. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
56. Yuan, Y., Huang, L., Guo, J., Zhang, C., Chen, X., and Wang, J. (2018). Ocnet: Object context network for scene parsing. arXiv.
57. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–23). Non-Local Neural Networks. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
58. Xie, S., Liu, S., Chen, Z., and Tu, Z. (2018, January 18–23). Attentional ShapeContextNet for Point Cloud Recognition. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00484 ↗
59. Yan, X., Zheng, C., Li, Z., Wang, S., and Cui, S. (2020, January 14–19). PointASNL: Robust Point Clouds Processing Using Nonlocal Neural Networks With Adaptive Sampling. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00563 ↗
60. Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. (2021, January 10–17). Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions. Proceedings of the IEEE International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00061 ↗
61. Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., and Chua, T.S. (2017, January 21–26). SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.667 ↗
62. Park, J., Woo, S., Lee, J.Y., and Kweon, I.S. (2018). Bam: Bottleneck attention module. arXiv.
63. Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., and Tang, X. (2017, January 21–26). Residual Attention Network for Image Classification. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.683 ↗
64. Liu, J.J., Hou, Q., Cheng, M.M., Wang, C., and Feng, J. (2020, January 14–19). Improving Convolutional Networks With Self-Calibrated Convolutions. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01011 ↗
65. Misra, D., Nalamada, T., Arasanipalai, A.U., and Hou, Q. (2021, January 3–8). Rotate to Attend: Convolutional Triplet Attention Module. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV48630.2021.00318 ↗
66. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 15–20). Dual Attention Network for Scene Segmentation. Proceedings of the IEEE Conference on Computer Vision Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00326 ↗
67. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Proceedings of the Neural Information Processing Systems, Montreal, QC, Canada.
68. Task-Aware Monocular Depth Estimation for 3D Object Detection
69. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Neural Information Processing Systems, Lake Tahoe, NV, USA.