Gaussian-UDSR: Real-Time Unbounded Dynamic Scene Reconstruction with 3D Gaussian Splatting
Source abstract
Abstract
Unbounded dynamic scene reconstruction is crucial for applications such as autonomous driving, robotics, and virtual reality. However, existing methods struggle to reconstruct dynamic scenes in unbounded outdoor environments due to challenges such as lighting variation, object motion, and sensor limitations, leading to inaccurate geometry and low rendering fidelity. In this paper, we proposed Gaussian-UDSR, a novel 3D Gaussian-based representation that efficiently reconstructs and renders high-quality, unbounded dynamic scenes in real time. Our approach fused LiDAR point clouds and Structure-from-Motion (SfM) point clouds obtained from an RGB camera, significantly improving depth estimation and geometric accuracy. To address dynamic appearance variations, we introduced a Gaussian color feature prediction network, which adaptively captures global and local feature information, enabling robust rendering under changing lighting conditions. Additionally, a pose-tracking mechanism ensured precise motion estimation for dynamic objects, enhancing realism and consistency. We evaluated Gaussian-UDSR on the Waymo and KITTI datasets, demonstrating state-of-the-art rendering quality with an 8.8% improvement in PSNR, a 75% reduction in LPIPS, and a fourfold speed improvement over existing methods. Our approach enables efficient, high-fidelity 3D reconstruction and fast real-time rendering of large-scale dynamic environments, while significantly reducing model storage overhead.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Computer Graphics and Visualization Techniques · Generative Adversarial Networks and Image Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Mechanical and Equipment Engineering, Hebei University of Engineering, Handan 056038, China
Key Laboratory of Intelligent Industrial Equipment Technology of Hebei Province, Handan 056038, China
Handan Key Laboratory of Intelligent Vehicles, Handan 056038, China
Institute of Automation Chinese Academy of Sciences, Beijing 100000, China
Jizhong Energy Fengfeng Group Co., Ltd., Handan 056200, China
Funding
National Natural Science Foundation of China · Natural Science Foundation of Hebei Province
References (48)
Show source-provided references
1. Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P.P., Barron, J.T., and Kretzschmar, H. (2022, January 18–24). Block-nerf: Scalable large scene neural view synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00807 ↗
2. Zhang, X., Kundu, A., Funkhouser, T., Guibas, L., Su, H., and Genova, K. (2023, January 17–24). Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00800 ↗
3. Ost, J., Laradji, I., Newell, A., Bahat, Y., and Heide, F. (2022, January 18–24). Neural point light fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01787 ↗
4. Xie, Z., Zhang, J., Li, W., Zhang, F., and Zhang, L. (2023). S-nerf: Neural radiance fields for street views. arXiv.
5. Guo, J., Deng, N., Li, X., Bai, Y., Shi, B., Wang, C., Ding, C., Wang, D., and Li, Y. (2023). Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv.
6. Lu, F., Xu, Y., Chen, G., Li, H., Lin, K.-Y., and Jiang, C. (2023, January 1–6). Urban radiance field representation with deformable neural mesh primitives. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00049 ↗
7. Rematas, K., Liu, A., Srinivasan, P.P., Barron, J.T., Tagliasacchi, A., Funkhouser, T., and Ferrari, V. (2022, January 18–24). Urban radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01259 ↗
8. Wu, Z., Liu, T., Luo, L., Zhong, Z., Chen, J., Xiao, H., Hou, C., Lou, H., Chen, Y., and Yang, R. (2023, January 22–23). Mars: An instance-aware, modular and realistic simulator for autonomous driving. Proceedings of the CAAI International Conference on Artificial Intelligence, Fuzhou, China.10.1007/978-981-99-8850-1_1 ↗
9. Ost, J., Mannan, F., Thuerey, N., Knodt, J., and Heide, F. (2021, January 20–25). Neural scene graphs for dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00288 ↗
10. Kundu, A., Genova, K., Yin, X., Fathi, A., Pantofaru, C., Guibas, L.J., Tagliasacchi, A., Dellaert, F., and Funkhouser, T. (2022, January 18–24). Panoptic neural fields: A semantic object-aware neural scene representation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01253 ↗
11. Yang, Z., Chen, Y., Wang, J., Manivasagam, S., Ma, W.-C., Yang, A.J., and Urtasun, R. (2023, January 17–24). Unisim: A neural closed-loop sensor simulator. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00140 ↗
12. Park, K., Sinha, U., Hedman, P., Barron, J.T., Bouaziz, S., Goldman, D.B., Martin-Brualla, R., and Seitz, S.M. (2021). Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv.10.1145/3478513.3480487 ↗
13. Fang, J., Zhou, D., Yan, F., Zhao, T., Zhang, F., Ma, Y., Wang, L., and Yang, R. (2020). Augmented LiDAR simulator for autonomous driving. arXiv.10.1109/LRA.2020.2969927 ↗
14. Wang, J., Manivasagam, S., Chen, Y., Yang, Z., Bârsan, I.A., Yang, A.J., Ma, W.-C., and Urtasun, R. (2023). Cadsim: Robust and scalable in-the-wild 3d reconstruction for controllable sensor simulation. arXiv.
15. Chen, Y., Rong, F., Duggal, S., Wang, S., Yan, X., Manivasagam, S., Xue, S., Yumer, E., and Urtasun, R. (2021, January 20–25). Geosim: Realistic video simulation via geometry-aware composition for self-driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00715 ↗
16. Manivasagam, S., Wang, S., Wong, K., Zeng, W., Sazanovich, M., Tan, S., Yang, B., Ma, W.-C., and Urtasun, R. (2020, January 13–19). Lidarsim: Realistic lidar simulation by leveraging the real world. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01118 ↗
17. Yang, Z., Manivasagam, S., Chen, Y., Wang, J., Hu, R., and Urtasun, R. (June, January 29). Reconstructing objects in-the-wild for realistic sensor simulation. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), London, UK.10.1109/ICRA48891.2023.10160535 ↗
18. Yang, Z., Chai, Y., Anguelov, D., Zhou, Y., Sun, P., Erhan, D., Rafferty, S., and Kretzschmar, H. (2020, January 13–19). Surfelgan: Synthesizing realistic sensor data for autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01113 ↗
19. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., and Ng, R. (2021). Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv.10.1007/978-3-030-58452-8_24 ↗
20. Garbin, S.J., Kowalski, M., Johnson, M., Shotton, J., and Valentin, J. (2021, January 10–17). Fastnerf: High-fidelity neural rendering at 200fps. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01408 ↗
21. Müller, T., Evans, A., Schied, C., and Keller, A. (2022). Instant neural graphics primitives with a multiresolution hash encoding. arXiv.10.1145/3528223.3530127 ↗
22. Reiser, C., Peng, S., Liao, Y., and Geiger, A. (2021, January 10–17). Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01407 ↗
23. Yu, A., Li, R., Tancik, M., Li, H., Ng, R., and Kanazawa, A. (2021, January 10–17). Plenoctrees for real-time rendering of neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00570 ↗
24. Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., and Kanazawa, A. (2022, January 18–24). Plenoxels: Radiance fields without neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00542 ↗
25. Deng, K., Liu, A., Zhu, J.-Y., and Ramanan, D. (2022, January 18–24). Depth-supervised nerf: Fewer views and faster training for free. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01254 ↗
26. Yang, J., Pavone, M., and Wang, Y. (2023, January 17–24). Freenerf: Improving few-shot neural rendering with free frequency regularization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00798 ↗
27. Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., and Srinivasan, P.P. (2021, January 10–17). Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00580 ↗
28. Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J.T., and Srinivasan, P.P. (2022, January 18–24). Ref-nerf: Structured view-dependent appearance for neural radiance fields. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00541 ↗
29. Niemeyer, M., Barron, J.T., Mildenhall, B., Sajjadi, M.S., Geiger, A., and Radwan, N. (2022, January 18–24). Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00540 ↗
30. Wang, G., Chen, Z., Loy, C.C., and Liu, Z. (2023, January 1–6). Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00832 ↗
31. Kerbl, B., Kopanas, G., Leimkühler, T., and Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. arXiv.10.1145/3592433 ↗
32. Wu, G., Yi, T., Fang, J., Xie, L., Zhang, X., Wei, W., Liu, W., Tian, Q., and Wang, X. (2024, January 16–22). 4d gaussian splatting for real-time dynamic scene rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01920 ↗
33. Shao, R., Sun, J., Peng, C., Zheng, Z., Zhou, B., Zhang, H., and Liu, Y. (2024, January 16–22). Control4d: Efficient 4d portrait editing with text. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00436 ↗
34. Chen, A., Xu, Z., Geiger, A., Yu, J., and Su, H. (2022). Tensorf: Tensorial radiance fields. Computer Vision—ECCV 2022, Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23–27 October 2022, Springer.10.1007/978-3-031-19824-3_20 ↗
35. Cao, A., and Johnson, J. (2023, January 17–24). Hexplane: A fast representation for dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00021 ↗
36. Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., and Jin, X. (2024, January 16–22). Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01922 ↗
37. Liang, Y., Khan, N., Li, Z., Nguyen-Phuoc, T., Lanman, D., Tompkin, J., and Xiao, L. (2023). Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. arXiv.
38. Huang, Y.-H., Sun, Y.-T., Yang, Z., Lyu, X., Cao, Y.-P., and Qi, X. (2024, January 16–22). Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00404 ↗
39. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III, Springer.10.1007/978-3-319-24574-4_28 ↗
40. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
41. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017). Automatic differentiation in pytorch. Advances in Neural Information Processing Systems 30, Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017, Neural Information Processing Systems Foundation, Inc. (NeurIPS).
42. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
43. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., and Caine, B. (2020, January 13–19). Scalability in perception for autonomous driving: Waymo open dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00252 ↗
44. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The kitti vision benchmark suite. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
45. Turki, H., Zhang, J.Y., Ferroni, F., and Ramanan, D. (2023, January 17–24). Suds: Scalable urban dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01191 ↗
46. Yang, J., Ivanovic, B., Litany, O., Weng, X., Kim, S.W., Li, B., Che, T., Xu, D., Fidler, S., and Pavone, M. (2023). Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. arXiv.
47. Chen, Y., Gu, C., Jiang, J., Zhu, X., and Zhang, L. (2023). Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering. arXiv.
48. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–23). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗