Semantic-Guided Spatial and Temporal Fusion Framework for Enhancing Monocular Video Depth Estimation
Source abstract
Abstract
Despite advancements in deep learning-based Monocular Depth Estimation (MDE), applying these models to video sequences remains challenging due to geometric ambiguities in texture-less regions and temporal instability caused by independent per-frame inference. To address these limitations, we propose STF-Depth, a novel post-processing framework that enhances depth quality by logically fusing heterogeneous information—geometric, semantic, and panoptic—without requiring additional retraining. Our approach introduces a robust RANSAC-based Vanishing Point Estimation to guide Dynamic Depth Gradient Correction for background separation, alongside Adaptive Instance Re-ordering to clarify occlusion relationships. Experimental results on the KITTI, NYU Depth V2, and TartanAir datasets demonstrate that STF-Depth functions as a universal plug-and-play module. Notably, it achieved a 25.7% reduction in Absolute Relative error (AbsRel) and significantly enhanced temporal consistency compared to state-of-the-art backbone models. These findings confirm the framework’s practicality for real-world applications requiring geometric precision and video stability, such as autonomous driving, robotics, and augmented reality (AR).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Video Coding and Compression Technologies · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Artificial Intelligence, Dongguk University, Seoul 04620, Republic of Korea
Funding
Ministry of Science and ICT
References (37)
Show source-provided references
1. Depth map prediction from a single image using a multi-scale deep network
2. Birkl, R., Wofk, D., and Müller, M. (2023). MiDaS v3.1—A Model Zoo for Robust Monocular Relative Depth Estimation. arXiv.
3. Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., and Zhao, H. (2024). Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. arXiv.10.1109/CVPR52733.2024.00987 ↗
4. Kim, H., and Son, Y. (2025). Generating Multi-View Action Data from a Monocular Camera Video by Fusing Human Mesh Recovery and 3D Scene Reconstruction. Appl. Sci., 15.10.3390/app151910372 ↗
5. Development of approach to an automated acquisition of static street view images using transformer architecture for analysis of Building characteristics10.1038/s41598-025-14786-3 ↗
6. Safety helmet monitoring on construction sites using YOLOv10 and advanced transformer architectures with surveillance and body-worn cameras10.1061/JCEMD4.COENG-16760 ↗
7. Watson, J., Aodha, O.M., Prisacariu, V., Brostow, G., and Firman, M. (2021). The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth. arXiv.10.1109/CVPR46437.2021.00122 ↗
8. Chen, S., Guo, H., Zhu, S., Zhang, F., Huang, Z., Feng, J., and Kang, B. (2025). Video Depth Anything: Consistent Depth Estimation for Super-Long Videos. arXiv.10.1109/CVPR52734.2025.02126 ↗
9. Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
10. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer10.1109/TPAMI.2020.3019967 ↗
11. Ranftl, R., Bochkovskiy, A., and Koltun, V. (2021, January 11–17). Vision transformers for dense prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01196 ↗
12. PSSCL: A progressive sample selection framework with contrastive loss designed for noisy labels10.1016/j.patcog.2024.111284 ↗
13. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
14. Kirillov, A., He, K., Girshick, R., Rother, C., and Dollár, P. (2019, January 15–20). Panoptic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00963 ↗
15. Jain, J., Li, J., Chiu, M.T., Hassani, A., Orlov, N., and Shi, H. (2023, January 18–22). Oneformer: One transformer to rule universal image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00292 ↗
16. Teed, Z., and Deng, J. (2020, January 23–28). Raft: Recurrent all-pairs field transforms for optical flow. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58536-5_24 ↗
17. Temporally consistent depth prediction with flow-guided memory units10.1109/TITS.2019.2942096 ↗
18. Watson, J., Mac Aodha, O., Prisacariu, V., Brostow, G., and Firman, M. (2021, January 19–25). The temporal opportunist: Self-supervised multi-frame monocular depth. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00122 ↗
19. Don’t forget the past: Recurrent depth estimation from monocular video10.1109/LRA.2020.3017478 ↗
20. Chen, S., Guo, H., Zhu, S., Zhang, F., Huang, Z., Feng, J., and Kang, B. (2025, January 11–15). Video Depth Anything: Consistent Depth Estimation for Super-Long Videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR52734.2025.02126 ↗
21. Consistent video depth estimation10.1145/3386569.3392377 ↗
22. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. (2021, January 10–17). Vivit: A video vision transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00676 ↗
23. Li, S., Luo, Y., Zhu, Y., Zhao, X., Li, Y., and Shan, Y. (2021, January 10–17). Enforcing temporal consistency in video depth estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00134 ↗
24. Wang, Y., Shi, M., Li, J., Huang, Z., Cao, Z., Zhang, J., Xian, K., and Lin, G. (2023, January 4–6). Neural video depth stabilizer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00868 ↗
25. Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. (2012, January 7–13). Indoor segmentation and support inference from rgbd images. Proceedings of the European conference on Computer Vision, Florence, Italy.10.1007/978-3-642-33715-4_54 ↗
26. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? the kitti vision benchmark suite. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
27. Wang, W., Zhu, D., Wang, X., Hu, Y., Qiu, Y., Wang, C., Hu, Y., Kapoor, A., and Scherer, S. (2020–24, January 24). TartanAir: A Dataset to Push the Limits of Visual SLAM. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341801 ↗
28. Depth anything v2
29. Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., and Zhao, H. (2024, January 16–22). Depth anything: Unleashing the power of large-scale unlabeled data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00987 ↗
30. Ke, B., Qu, K., Wang, T., Metzger, N., Huang, S., Li, B., Obukhov, A., and Schindler, K. (2025). Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis. arXiv.10.1109/TPAMI.2025.3591076 ↗
31. Bhat, S.F., Birkl, R., Wofk, D., Wonka, P., and Müller, M. (2023). Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv.
32. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
33. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography10.1145/358669.358692 ↗
34. Kim, D., Ka, W., Ahn, P., Joo, D., Chun, S., and Kim, J. (2022). Global-local path networks for monocular depth estimation with vertical cutdepth. arXiv.
35. Lai, W.S., Huang, J.B., Wang, O., Shechtman, E., Yumer, E., and Yang, M.H. (2018, January 8–14). Learning blind video temporal consistency. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01267-0_11 ↗
36. Yang, S., Lee, M., Cho, S., Lee, J., and Lee, S. (2024). STATIC: Surface Temporal Affine for TIme Consistency in Video Monocular Depth Estimation. arXiv.
37. Yasarla, R., Singh, M.K., Cai, H., Shi, Y., Jeong, J., Zhu, Y., Han, S., Garrepalli, R., and Porikli, F. (October, January 29). Futuredepth: Learning to predict the future improves video depth estimation. Proceedings of the European Conference on Computer Vision, Milan, Italy.