Dynamic Street-Scene Reconstruction with Semantic Priors and Temporal Constraints
Source abstract
Abstract
Dynamic street-scene reconstruction from sparse viewpoints over long temporal spans is challenged by temporal instability, ghosting near occlusions, and background drift. This paper presents SPT-Gauss (Semantic Prior and Temporal constraint-enhanced Gaussian splatting), a Gaussian-splatting framework that improves dynamic reconstruction without object-level annotations by combining dense semantic priors with lightweight, parameter-level temporal regularization. SPT-Gauss distills per-pixel semantic features from a frozen 2D foundation model into 4D Gaussian primitives, estimates static and dynamic regions via a dual-evidence motion mask, and regularizes temporal parameters through a semantic-guided velocity constraint and a static-lifetime prior to suppress spurious background motion. Experiments on the Waymo Open Dataset and KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) show consistent improvements over representative baselines in both 4D reconstruction and novel-view synthesis, with reduced temporal artifacts and improved fidelity in motion-challenging regions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Robotics and Sensor-Based Localization · 3D Shape Modeling and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Technology, ShanghaiTech University, Shanghai 201210, China
Shanghai Institute of Technical Physics, Chinese Academy of Sciences, Shanghai 200083, China
University of Chinese Academy of Sciences, Beijing 100049, China
References (45)
Show source-provided references
1. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., and Ng, R. (2020, January 23–28). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58452-8_24 ↗
2. Müller, T., Evans, A., Schied, C., and Keller, A. (2022). Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph., 41.10.1145/3528223.3530127 ↗
3. Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., and Kanazawa, A. (2022, January 19–20). Plenoxels: Radiance Fields without Neural Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00542 ↗
4. Chen, A., Xu, Z., Geiger, A., Yu, J., and Su, H. (2022, January 23–27). TensoRF: Tensorial Radiance Fields. Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel.10.1007/978-3-031-19824-3_20 ↗
5. Fridovich-Keil, S., Meanti, G., Warburg, F.R., Recht, B., and Kanazawa, A. (2023, January 18–22). K-Planes: Explicit Radiance Fields in Space, Time, and Appearance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01201 ↗
6. 3D Gaussian Splatting for Real-Time Radiance Field Rendering10.1145/3592433 ↗
7. Luiten, J., Kopanas, G., Leibe, B., and Ramanan, D. (2024, January 18–21). Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis. Proceedings of the International Conference on 3D Vision (3DV), Davos, Switzerland.10.1109/3DV62453.2024.00044 ↗
8. Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., and Jin, X. (2024, January 17–18). Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01922 ↗
9. Bae, J., Kim, S., Yun, Y., Lee, H., Bang, G., and Uh, Y. (October, January 29). Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.
10. Wan, D., Lu, R., and Zeng, G. (2024, January 21–27). Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction. Proceedings of the 41st International Conference on Machine Learning (ICML), Vienna, Austria.
11. Zhu, R., Liang, Y., Chang, H., Deng, J., Lu, J., Yang, W., Zhang, T., and Zhang, Y. (2024, January 9–15). MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
12. Wu, G., Yi, T., Fang, J., Xie, L., Zhang, X., Wei, W., Liu, W., Tian, Q., and Wang, X. (2024, January 17–21). 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01920 ↗
13. Lin, Y., Dai, Z., Zhu, S., and Yao, Y. (2024, January 17–21). Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01997 ↗
14. Duan, Y., Wei, F., Dai, Q., He, Y., Chen, W., and Chen, B. (August, January 27). 4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes. Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference (ACM SIGGRAPH 2024 Conference Papers), Denver, CO, USA.10.1145/3641519.3657463 ↗
15. Huang, N., Wei, X., Zheng, W., An, P., Lu, M., Zhan, W., Tomizuka, M., Keutzer, K., and Zhang, S. (2026, January 1–5). S3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Vienna, Austria. (accepted, to appear).
16. Lan, L., Shao, T., Lu, Z., Zhang, Y., Jiang, C., and Yang, Y. (2025, January 10–14). 3DGS2: Near Second-order Converging 3D Gaussian Splatting. Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference (ACM SIGGRAPH Conference Papers’25), Vancouver, BC, Canada.10.1145/3721238.3730687 ↗
17. Chen, Y., Gu, C., Jiang, J., Zhu, X., and Zhang, L. (2023). Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-Time Rendering. arXiv.
18. SplatFlow: Learning Multi-frame Optical Flow via Splatting10.1007/s11263-024-01993-0 ↗
19. Mou, L., Chen, J.K., and Wang, Y.X. (2024). Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion. arXiv.10.1109/CVPR52733.2024.01907 ↗
20. Zhou, X., Lin, Z., Shan, X., Wang, Y., Sun, D., and Yang, M.H. (2024, January 17–21). DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.02044 ↗
21. Yan, Y., Lin, H., Zhou, C., Wang, W., Sun, H., Zhan, K., Lang, X., Zhou, X., and Peng, S. (October, January 29). Street Gaussians for Modeling Dynamic Urban Scenes. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.
22. Chen, Z., Yang, J., Huang, J., de Lutio, R., Esturo, J.M., Ivanovic, B., Litany, O., Gojcic, Z., Fidler, S., and Pavone, M. (2025, January 24–28). OmniRe: Omni Urban Scene Reconstruction. Proceedings of the International Conference on Learning Representations (ICLR), Singapore.
23. Liu, Y., Luo, C., Fan, L., Wang, N., Peng, J., and Zhang, Z. (2024). CityGaussian: Real-Time High-Quality Large-Scale Scene Rendering with Gaussians. Computer Vision—ECCV 2024, Milan, Italy, 29 September–4 October, Springer Nature.10.1007/978-3-031-72640-8_15 ↗
24. Kim, H., Cho, J., and Kang, S.J. (2024, January 17–18). Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.
25. Peng, C., Zhang, C., Wang, Y., Xu, C., Xie, Y., Zheng, W., Keutzer, K., Tomizuka, M., and Zhan, W. (2024). DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenes. arXiv.10.1109/CVPR52734.2025.00636 ↗
26. Song, R., Liang, C., Xia, Y., Zimmer, W., Cao, H., Caesar, H., Festag, A., and Knoll, A. (2025, January 19–23). Coda-4dgs: Dynamic gaussian splatting with context and deformation awareness for autonomous driving. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Honolulu, HI, USA.
27. Yang, J., Ivanovic, B., Litany, O., Weng, X., Kim, S.W., Li, B., Che, T., Xu, D., Fidler, S., and Pavone, M. (2024, January 7–11). EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria.
28. Zhou, S., Chang, H., Jiang, S., Fan, Z., Zhu, Z., Xu, D., Chari, P., You, S., Wang, Z., and Kadambi, A. (2024, January 17–18). Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.02048 ↗
29. Chen, G., and Wang, W. (2024). A Survey on 3D Gaussian Splatting. arXiv.
30. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the International Conference on Machine Learning (ICML), Online.
31. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., and El-Nouby, A. (2023). DINOv2: Learning Robust Visual Features without Supervision. arXiv.
32. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023, January 2–3). Segment Anything. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00371 ↗
33. Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., and Ranftl, R. (2022, January 25–29). Language-driven Semantic Segmentation. Proceedings of the International Conference on Learning Representations (ICLR), Online.
34. Cheng, B., Schwing, A.G., and Kirillov, A. (2021, January 6–14). Per-Pixel Classification is Not All You Need for Semantic Segmentation. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Online.
35. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., and Girdhar, R. (2022, January 18–24). Masked-attention Mask Transformer for Universal Image Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00135 ↗
36. Compositing Digital Images10.1145/964965.808606 ↗
37. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
38. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., and Caine, B. (2020, January 14–19). Scalability in Perception for Autonomous Driving: Waymo Open Dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00252 ↗
39. Turki, H., Zhang, J.Y., Ferroni, F., and Ramanan, D. (2023, January 17–24). SUDS: Scalable Urban Dynamic Scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01191 ↗
40. Guo, J., Deng, N., Li, X., Bai, Y., Shi, B., Wang, C., Ding, C., Wang, D., and Li, Y. (2023). StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Views. arXiv.
41. Wu, Z., Liu, T., Luo, L., Zhong, Z., Chen, J., Xiao, H., Hou, C., Lou, H., Chen, Y., and Yang, R. (2023, January 22–23). MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving. Proceedings of the CAAI International Conference on Artificial Intelligence (CICAI), Fuzhou, China.10.1007/978-981-99-8850-1_1 ↗
42. Image Quality Assessment: From Error Visibility to Structural Similarity10.1109/TIP.2003.819861 ↗
43. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18–22). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00068 ↗
44. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
45. Loshchilov, I., and Hutter, F. (2017, January 24–26). SGDR: Stochastic Gradient Descent with Warm Restarts. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France.