Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images
Source abstract
Abstract
Semantic modeling is a challenging task that has received widespread attention in recent years. With the help of mini Unmanned Aerial Vehicles (UAVs), multi-view high-resolution aerial images of large-scale scenes can be conveniently collected. In this paper, we propose a semantic Multi-View Stereo (MVS) method to reconstruct 3D semantic models from 2D images. Firstly, 2D semantic probability distribution is obtained by Convolutional Neural Network (CNN). Secondly, the calibrated cameras poses are determined by Structure from Motion (SfM), while the depth maps are estimated by learning MVS. Combining 2D segmentation and 3D geometry information, dense point clouds with semantic labels are generated by a probability-based semantic fusion method. In the final stage, the coarse 3D semantic point cloud is optimized by both local and global refinements. By making full use of the multi-view consistency, the proposed method efficiently produces a fine-level 3D semantic point cloud. The experimental result evaluated by re-projection maps achieves 88.4% Pixel Accuracy on the Urban Drone Dataset (UDD). In conclusion, our graph-based semantic fusion procedure and refinement based on local and global information can suppress and reduce the re-projection error.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Remote Sensing and LiDAR Applications · 3D Surveying and Cultural Heritage
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graphics & Interaction Lab, School of Electronics Engineering and Computer Sciences, Peking University, Beijing 100871, China
Key Lab of Machine Perception and Intelligent, MOE, Department of Computer Sciences, Peking University, Beijing 100871, China
Beijing Engineering Technology Research Center of Virtual Simulation and Visualization, Peking University, Beijing 100871, China
Funding
The National Key Technology Research and Development Program of China · National Natural Science Foundation of China (NSFC) · Equipment Development Project
References (34)
Show source-provided references
1. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
2. Segnet: A deep convolutional encoder-decoder architecture for image segmentation10.1109/TPAMI.2016.2644615 ↗
3. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
4. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid scene parsing network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
5. Yao, Y., Luo, Z., Li, S., Fang, T., and Quan, L. (2018, January 8–14). Mvsnet: Depth inference for unstructured multi-view stereo. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01237-3_47 ↗
6. Yao, Y., Luo, Z., Li, S., Shen, T., Fang, T., and Quan, L. (2019, January 16–20). Recurrent mvsnet for high-resolution multi-view stereo depth inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00567 ↗
7. Chen, Y., Wang, Y., Lu, P., Chen, Y., and Wang, G. (2018, January 23–26). Large-scale structure from motion with semantic constraints of aerial images. Proceedings of the Chinese Conference on Pattern Recognition and Computer Vision (PRCV), Guangzhou, China.10.1007/978-3-030-03398-9_30 ↗
8. Bao, S.Y., and Savarese, S. (2011, January 20–25). Semantic structure from motion. Proceedings of the CVPR 2011, Providence, RI, USA.10.1109/CVPR.2011.5995462 ↗
9. Martinovic, A., Knopp, J., Riemenschneider, H., and Van Gool, L. (2015, January 7–13). 3D all the way: Semantic segmentation of urban scenes from start to end in 3d. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Santiago, Chile.10.1109/CVPR.2015.7299075 ↗
10. Wolf, D., Prankl, J., and Vincze, M. (2015, January 26–30). Fast semantic segmentation of 3D point clouds using a dense CRF with learned parameters. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.10.1109/ICRA.2015.7139875 ↗
11. Dense semantic 3d reconstruction
12. Hane, C., Zach, C., Cohen, A., Angst, R., and Pollefeys, M. (2013, January 23–28). Joint 3D scene reconstruction and class segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.20 ↗
13. Savinov, N., Ladicky, L., Hane, C., and Pollefeys, M. (2015, January 7–13). Discrete optimization of ray potentials for semantic 3d reconstruction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Santiago, Chile.10.1109/CVPR.2015.7299190 ↗
14. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. (2015, January 7–13). 3d shapenets: A deep representation for volumetric shapes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Santiago, Chile.
15. Maturana, D., and Scherer, S. (October, January 28). Voxnet: A 3d convolutional neural network for real-time object recognition. Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.10.1109/IROS.2015.7353481 ↗
16. Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21–26). Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.
17. Qi, C.R., Yi, L., Su, H., and Guibas, L.J. (2017, January 4–9). Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
18. Vineet, V., Miksik, O., Lidegaard, M., Nießner, M., Golodetz, S., Prisacariu, V.A., Kähler, O., Murray, D.W., Izadi, S., and Pérez, P. (2015, January 26–30). Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.10.1109/ICRA.2015.7138983 ↗
19. Zhao, C., Sun, L., and Stolkin, R. (2017, January 10–12). A fully end-to-end deep learning approach for real-time simultaneous 3D reconstruction and material recognition. Proceedings of the 2017 18th International Conference on Advanced Robotics (ICAR), Hong Kong, China.10.1109/ICAR.2017.8023499 ↗
20. McCormac, J., Handa, A., Davison, A., and Leutenegger, S. (June, January 29). Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, Singapore.10.1109/ICRA.2017.7989538 ↗
21. Li, X., Wang, D., Ao, H., Belaroussi, R., and Gruyer, D. (2019). Fast 3D Semantic Mapping in Road Scenes. Appl. Sci., 9.10.20944/preprints201901.0009.v1 ↗
22. Zhou, Y., Shen, S., and Hu, Z. (2018, January 5–8). Fine-level semantic labeling of large-scale 3d model by active learning. Proceedings of the 2018 International Conference on 3D Vision (3DV), Verona, Italy.10.1109/3DV.2018.00066 ↗
23. Semantic photogrammetry: boosting image-based 3D reconstruction with semantic labeling
24. Fusion of images and point clouds for the semantic segmentation of large-scale 3D scenes based on deep learning10.1016/j.isprsjprs.2018.04.022 ↗
25. Schonberger, J.L., and Frahm, J.M. (2016, January 27–30). Structure-from-motion revisited. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.445 ↗
26. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
27. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–8). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
28. Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., and Aanæs, H. (2014, January 23–28). Large scale multi-view stereopsis evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.59 ↗
29. Galliani, S., Lasinger, K., and Schindler, K. (2015, January 7–13). Massively parallel multiview stereopsis by surface normal diffusion. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.106 ↗
30. Sedlacek, D., and Zara, J. (December, January 30). Graph cut based point-cloud segmentation for polygonal reconstruction. Proceedings of the International Symposium on Visual Computing, Las Vegas, NV, USA.
31. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision10.1109/TPAMI.2004.60 ↗
32. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
33. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2015). TensorFlow: Large-scale machine learning on heterogeneous systems. arXiv.
34. Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. Proceedings of COMPSTAT’2010;, Springer.10.1007/978-3-7908-2604-3_16 ↗