KD-PatchMatch: A Self-Supervised Training Learning-Based PatchMatch
Source abstract
Abstract
Traditional learning-based multi-view stereo (MVS) methods usually need to find the correct depth value from a large number of depth candidates, which leads to huge memory consumption and slow inference. To address these problems, we propose a probabilistic depth sampling in the learning-based PatchMatch framework, i.e., sampling a small number of depth candidates from a single-view probability distribution, which achieves the purpose of saving computational resources. Furthermore, to overcome the difficulty of obtaining ground-truth depth for outdoor large-scale scenes, we also propose a self-supervised training pipeline based on knowledge distillation, which involves self-supervised teacher training and student training based on knowledge distillation. Extensive experiments show that our approach outperforms other recent learning-based MVS methods on DTU, Tanks and Temples, and ETH3D datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Image Processing Techniques and Applications · Optical measurement and interference techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China
Funding
The National Natural Science Foundation of China
References (44)
Show source-provided references
1. Wang, R., Schworer, M., and Cremers, D. (2017, January 22–29). Stereo DSO: Large-scale direct sparse visual odometry with stereo cameras. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.421 ↗
2. Duggal, S., Wang, S., Ma, W.C., Hu, R., and Urtasun, R. (November, January 27). Deeppruner: Learning efficient stereo matching via differentiable patchmatch. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
3. Geometry-and appearance-based reasoning of construction progress monitoring10.1061/(ASCE)CO.1943-7862.0001428 ↗
4. EMVS: Event-based multi-view stereo—3D reconstruction with an event camera in real-time10.1007/s11263-017-1050-6 ↗
5. Luo, K., Guan, T., Ju, L., Huang, H., and Luo, Y. (November, January 27). P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
6. Xu, Q., and Tao, W. (2020, January 7–12). Learning inverse depth regression for multi-view stereo with correlation cost volume. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6939 ↗
7. Yao, Y., Luo, Z., Li, S., Fang, T., and Quan, L. (2018, January 8–14). Mvsnet: Depth inference for unstructured multi-view stereo. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01237-3_47 ↗
8. Yao, Y., Luo, Z., Li, S., Shen, T., Fang, T., and Quan, L. (2019, January 16–17). Recurrent mvsnet for high-resolution multi-view stereo depth inference. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00567 ↗
9. Large-scale data for multiple-view stereopsis10.1007/s11263-016-0902-9 ↗
10. Tanks and temples: Benchmarking large-scale scene reconstruction10.1145/3072959.3073599 ↗
11. Schops, T., Schonberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., and Geiger, A. (2017, January 21–26). A multi-view stereo benchmark with high-resolution images and multi-camera videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.272 ↗
12. Gu, X., Fan, Z., Zhu, S., Dai, Z., Tan, F., and Tan, P. (2020, January 13–19). Cascade cost volume for high-resolution multi-view stereo and stereo matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00257 ↗
13. Zhang, J., Yao, Y., Li, S., Luo, Z., and Fang, T. (2020). Visibility-aware multi-view stereo network. arXiv.
14. Yang, J., Mao, W., Alvarez, J.M., and Liu, M. (2020, January 13–19). Cost volume pyramid based depth inference for multi-view stereo. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00493 ↗
15. Wang, F., Galliani, S., Vogel, C., Speciale, P., and Pollefeys, M. (2021, January 20–25). Patchmatchnet: Learned multi-view patchmatch stereo. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01397 ↗
16. Chen, R., Han, S., Xu, J., and Su, H. (November, January 27). Point-based multi-view stereo network. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
17. Lee, J.Y., DeGol, J., Zou, C., and Hoiem, D. (2021, January 11–17). Patchmatch-rl: Deep mvs with pixelwise depth, normal, and visibility. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00610 ↗
18. Cheng, S., Xu, Z., Zhu, S., Li, Z., Li, L.E., Ramamoorthi, R., and Su, H. (2020, January 13–19). Deep stereo using adaptive thin volume representation with uncertainty awareness. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00260 ↗
19. A theory of shape by space carving10.1023/A:1008191222954 ↗
20. Photorealistic scene reconstruction by voxel coloring10.1023/A:1008176507526 ↗
21. Accurate, dense, and robust multiview stereopsis10.1109/TPAMI.2009.161 ↗
22. A quasi-dense approach to surface reconstruction from uncalibrated images10.1109/TPAMI.2005.44 ↗
23. Galliani, S., Lasinger, K., and Schindler, K. (2015, January 7–13). Massively parallel multiview stereopsis by surface normal diffusion. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.106 ↗
24. Schonberger, J.L., and Frahm, J.M. (2016, January 27–30). Structure-from-motion revisited. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.445 ↗
25. Xu, Q., and Tao, W. (2019, January 16–17). Multi-scale geometric consistency guided multi-view stereo. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00563 ↗
26. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
27. Mirzadeh, S.I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., and Ghasemzadeh, H. (2020, January 7–12). Improved knowledge distillation via teacher assistant. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i04.5963 ↗
28. Li, T., Li, J., Liu, Z., and Zhang, C. (2020, January 13–19). Few sample knowledge distillation for efficient network compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01465 ↗
29. Probabilistic knowledge transfer for lightweight deep representation learning10.1109/TNNLS.2020.2995884 ↗
30. Charrow, B., Liu, S., Kumar, V., and Michael, N. (2015, January 26–30). Information-theoretic mapping using cauchy-schwarz quadratic mutual information. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.10.1109/ICRA.2015.7139865 ↗
31. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
32. Bae, G., Budvytis, I., and Cipolla, R. (2022, January 18–24). Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00286 ↗
33. Tan, M., and Le, Q. (2019, January 9–15). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
34. What uncertainties do we need in bayesian deep learning for computer vision?
35. Ding, Y., Zhu, Q., Liu, X., Yuan, W., Zhang, H., and Zhang, C. (2022, January 23–27). KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view Stereo. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19821-2_36 ↗
36. Khot, T., Agrawal, S., Tulsiani, S., Mertz, C., Lucey, S., and Hebert, M. (2019). Learning unsupervised multi-view stereopsis via robust photometric consistency. arXiv.
37. Tung, F., and Mori, G. (November, January 27). Similarity-preserving knowledge distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
38. Ji, M., Gall, J., Zheng, H., Liu, Y., and Fang, L. (2017, January 22–29). Surfacenet: An end-to-end 3d neural network for multiview stereopsis. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.253 ↗
39. Campbell, N.D., Vogiatzis, G., Hernández, C., and Cipolla, R. (2008, January 12–18). Using multiple hypotheses to improve depth-maps for multi-view stereo. Proceedings of the European Conference on Computer Vision, Marseille, France.10.1007/978-3-540-88682-2_58 ↗
40. Efficient large-scale multi-view stereo for ultra high-resolution image sets10.1007/s00138-011-0346-8 ↗
41. Yu, Z., and Gao, S. (2020, January 13–19). Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00202 ↗
42. Moulon, P., Monasse, P., Perrot, R., and Marlet, R. (2016, January 4). Openmvg: Open multiple view geometry. Proceedings of the International Workshop on Reproducible Research in Pattern Recognition, Cancún, Mexico.10.1007/978-3-319-56414-2_5 ↗
43. Xu, Q., and Tao, W. (2020). Pvsnet: Pixelwise visibility-aware multi-view stereo network. arXiv.
44. Fuhrmann, S., Langguth, F., and Goesele, M. (2014). Mve-a Multi-View Reconstruction Environment, GCH.10.1016/j.cag.2015.09.003 ↗