RE-PU: A Self-Supervised Arbitrary-Scale Point Cloud Upsampling Method Based on Reconstruction
Source abstract
Abstract
The point clouds obtained directly from three-dimensional scanning devices are often sparse and noisy. Therefore, point cloud upsampling plays an increasingly crucial role in fields such as point cloud reconstruction and rendering. However, point cloud upsampling methods are primarily supervised and fixed-rate, which restricts their applicability in various scenarios. In this paper, we propose a novel point cloud upsampling method, named RE-PU, which is based on the point cloud reconstruction and achieves self-supervised upsampling at arbitrary rates. The proposed method consists of two main stages: the first stage is to train a network to reconstruct the original point cloud from a prior distribution, and the second stage is to upsample the point cloud data by increasing the number of sampled points on the prior distribution with the trained model. The experimental results demonstrate that the proposed method can achieve comparable outcomes to supervised methods in terms of both visual quality and quantitative metrics.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: 3D Shape Modeling and Analysis · Computer Graphics and Visualization Techniques · Advanced Vision and Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer and Electronics Information, Guangxi University, Nanning 530004, China
Guangxi Key Laboratory of Multimedia Communications and Network Technology, Nanning 530004, China
Funding
National Natural Science Foundation of China
References (56)
Show source-provided references
1. Luo, L., Tang, L., Zhou, W., Wang, S., and Yang, Z.X. (2021, January 10–17). Pu-eva: An edge-vector based approximation solution for flexible-scale point cloud upsampling. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01590 ↗
2. Liu, Y., Wang, Y., and Liu, Y. (2022, January 13–16). Refine-PU: A Graph Convolutional Point Cloud Upsampling Network using Spatial Refinement. Proceedings of the 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP), Suzhou, China.10.1109/VCIP56404.2022.10008811 ↗
3. Li, T., Lin, Y., Cheng, B., Ai, G., Yang, J., and Fang, L. (2024). PU-CTG: A Point Cloud Upsampling Network Using Transformer Fusion and GRU Correction. Remote. Sens., 16.10.3390/rs16030450 ↗
4. Pu-dense: Sparse tensor-based point cloud geometry upsampling10.1109/TIP.2022.3180904 ↗
5. Edge-aware point set resampling
6. Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21–26). Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.
7. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
8. Du, H., Yan, X., Wang, J., Xie, D., and Pu, S. (2022, January 4–8). Point cloud upsampling via cascaded refinement network. Proceedings of the Asian Conference on Computer Vision, Macao, China.10.1007/978-3-031-26319-4_7 ↗
9. Qiu, S., Anwar, S., and Barnes, N. (2022, January 4–8). Pu-transformer: Point cloud upsampling transformer. Proceedings of the Asian Conference on Computer Vision, Macao, China.10.1007/978-3-031-26319-4_20 ↗
10. PU-Ray: Domain-Independent Point Cloud Upsampling via Ray Marching on Neural Implicit Surface10.1109/TITS.2024.3489574 ↗
11. Dynamic graph cnn for learning on point clouds
12. Pct: Point cloud transformer10.1007/s41095-021-0229-5 ↗
13. Zhang, Y., Zhao, W., Sun, B., Zhang, Y., and Wen, W. (2022). Point cloud upsampling algorithm: A systematic review. Algorithms, 15.10.3390/a15040124 ↗
14. Yu, L., Li, X., Fu, C.W., Cohen-Or, D., and Heng, P.A. (2018, January 18–23). Pu-net: Point cloud upsampling network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00295 ↗
15. Pointnet++: Deep hierarchical feature learning on point sets in a metric space
16. Yifan, W., Wu, S., Huang, H., Cohen-Or, D., and Sorkine-Hornung, O. (2019, January 15–20). Patch-based progressive 3D point set upsampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00611 ↗
17. Qian, G., Abualshour, A., Li, G., Thabet, A., and Ghanem, B. (2021, January 20–25). Pu-gcn: Point cloud upsampling using graph convolutional networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01151 ↗
18. Li, R., Li, X., Heng, P.A., and Fu, C.W. (2021, January 20–25). Point cloud upsampling via disentangled refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00041 ↗
19. Long, C., Zhang, W., Li, R., Wang, H., Dong, Z., and Yang, B. (2022, January 10–14). Pc2-pu: Patch correlation and point correlation for effective point cloud upsampling. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3547777 ↗
20. Wang, J., Chen, J., Shi, Y., Ling, N., and Yin, B. (November, January 29). SSPU-Net: A Structure Sensitive Point Cloud Upsampling Network with Multi-Scale Spatial Refinement. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
21. Zhao, W., Zhang, H., Zheng, C., Yan, X., Cui, S., and Li, Z. (November, January 29). CPU: Codebook Lookup Transformer with Knowledge Distillation for Point Cloud Upsampling. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
22. Cai, P., Wu, Z., Wu, X., and Wang, S. (2023). Parametric Surface Constrained Upsampler Network for Point Cloud. arXiv.10.1609/aaai.v37i1.25097 ↗
23. Qian, Y., Hou, J., Kwong, S., and He, Y. (2020, January 23–28). PUGeo-Net: A geometry-centric network for 3D point cloud upsampling. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58529-7_44 ↗
24. Deep magnification-flexible upsampling over 3d point clouds10.1109/TIP.2021.3115385 ↗
25. Li, R., Li, X., Fu, C.W., Cohen-Or, D., and Heng, P.A. (November, January 21). Pu-gan: A point cloud upsampling adversarial network. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
26. Pufa-gan: A frequency-aware generative adversarial network for 3d point cloud upsampling10.1109/TIP.2022.3222918 ↗
27. Zhou, K., Dong, M., and Arslanturk, S. (2022, January 18–22). “Zero-Shot” Point Cloud Upsampling. Proceedings of the 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan.10.1109/ICME52920.2022.9859662 ↗
28. Kumbar, A., Anvekar, T., Tabib, R.A., and Mudenagudi, U. (2023, January 2–6). ASUR3D: Arbitrary Scale Upsampling and Refinement of 3D Point Clouds using Local Occupancy Fields. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Paris, France.10.1109/ICCVW60793.2023.00180 ↗
29. Kumbar, A., Anvekar, T., Vikrama, T.A., Tabib, R.A., and Mudenagudi, U. (2023, January 2–6). TP-NoDe: Topology-aware Progressive Noising and Denoising of Point Clouds towards Upsampling. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCVW60793.2023.00241 ↗
30. Zhao, Y., Hui, L., and Xie, J. (2021, January 20–24). Sspu-net: Self-supervised point cloud upsampling via differentiable rendering. Proceedings of the 29th ACM International Conference on Multimedia, Virtual.10.1145/3474085.3475381 ↗
31. Zhao, W., Liu, X., Zhong, Z., Jiang, J., Gao, W., Li, G., and Ji, X. (2022, January 18–24). Self-supervised arbitrary-scale point clouds upsampling via implicit neural representation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00204 ↗
32. Spu-net: Self-supervised point cloud upsampling by coarse-to-fine reconstruction with self-projection optimization10.1109/TIP.2022.3182266 ↗
33. Hu, X., Mu, H., Zhang, X., Wang, Z., Tan, T., and Sun, J. (2019, January 15–20). Meta-SR: A magnification-arbitrary network for super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00167 ↗
34. Meta-PU: An arbitrary-scale upsampling network for point cloud10.1109/TVCG.2021.3058311 ↗
35. Feng, W., Li, J., Cai, H., Luo, X., and Zhang, J. (2022, January 18–24). Neural points: Point cloud representation with neural fields for arbitrary upsampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01808 ↗
36. PU-Flow: A point cloud upsampling network with normalizing flows10.1109/TVCG.2022.3196334 ↗
37. Mao, A., Duan, Y., Wen, Y.H., Du, Z., Cai, H., and Liu, Y.J. (2023, January 19–25). Invertible residual neural networks with conditional injector and interpolator for point cloud upsampling. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, Macao, China.10.24963/ijcai.2023/141 ↗
38. He, Y., Tang, D., Zhang, Y., Xue, X., and Fu, Y. (2023, January 17–24). Grad-PU: Arbitrary-Scale Point Cloud Upsampling via Gradient Descent with Learned Distance Functions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00518 ↗
39. Unsupervised point cloud representation learning with deep neural networks: A survey10.1109/TPAMI.2023.3262786 ↗
40. Girdhar, R., Fouhey, D.F., Rodriguez, M., and Gupta, A. (2016, January 11–14). Learning a predictable and generative vector representation for objects. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part VI 14.10.1007/978-3-319-46466-4_29 ↗
41. Yang, Y., Feng, C., Shen, Y., and Tian, D. (2018, January 18–23). Foldingnet: Point cloud auto-encoder via deep grid deformation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00029 ↗
42. Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., and Aubry, M. (2018, January 18–23). A papier-mâché approach to learning 3D surface generation. Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00030 ↗
43. Liu, X., Han, Z., Wen, X., Liu, Y.S., and Zwicker, M. (2019, January 21–25). L2g auto-encoder: Understanding point clouds by local-to-global reconstruction with hierarchical self-attention. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3350960 ↗
44. Zhao, Y., Birdal, T., Deng, H., and Tombari, F. (2019, January 15–20). 3D point capsule networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00110 ↗
45. Deep unsupervised learning of 3D point clouds via graph topology inference and filtering10.1109/TIP.2019.2957935 ↗
46. Gao, X., Hu, W., and Qi, G.J. (2020, January 13–19). Graphter: Unsupervised learning of graph transformation equivariant representations via auto-encoding node-wise transformations. Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00719 ↗
47. Eckart, B., Yuan, W., Liu, C., and Kautz, J. (2021, January 20–25). Self-supervised learning on 3D point clouds by learning discrete generative models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00815 ↗
48. Pang, Y., Wang, W., Tay, F.E., Liu, W., Tian, Y., and Yuan, L. (2022, January 23–27). Masked autoencoders for point cloud self-supervised learning. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20086-1_35 ↗
49. Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training
50. Park, J.J., Florence, P., Straub, J., Newcombe, R., and Lovegrove, S. (2019, January 15–20). Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00025 ↗
51. Ma, B., Han, Z., Liu, Y.S., and Zwicker, M. (2021, January 18–24). Neural-Pull: Learning Signed Distance Function from Point clouds by Learning to Pull Space onto Surface. Proceedings of the International Conference on Machine Learning. PMLR, Virtual.
52. Chen, Z., and Zhang, H. (2019, January 15–20). Learning implicit fields for generative shape modeling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00609 ↗
53. Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., and Geiger, A. (2019, January 15–20). Occupancy networks: Learning 3d reconstruction in function space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00459 ↗
54. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany. Proceedings, Part III 18.
55. Zhao, H., Jiang, L., Jia, J., Torr, P.H., and Koltun, V. (2021, January 10–17). Point transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01595 ↗
56. Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., and Su, H. (2015). Shapenet: An information-rich 3D model repository. arXiv.