Three-Dimensional Gaussian Style Transfer Method Based on Two-Dimensional Priors and Iterative Optimization
Source abstract
Abstract
To address the limitations of existing optimization-based 3D style transfer methods in terms of visual quality, 3D consistency, and real-time rendering performance, we propose a novel 3D Gaussian scene style transfer method based on 2D priors and iterative optimization. Our approach introduces a progressive training pipeline that alternates between fine-tuning the 3D Gaussian field and updating a set of supervised stylized images. By gradually injecting style information into the 3D scene through iterative refinement, the method effectively preserves the geometric structure and spatial coherence across viewpoints. Furthermore, we incorporated a pre-trained stable diffusion model as a 2D prior to guide the style adaptation of the 3D Gaussian representation. The combination of diffusion priors and differentiable 3D Gaussian rendering enables high-fidelity style transfer while maintaining real-time rendering capability. Extensive experiments demonstrate that our method significantly improves the visual quality and multi-view consistency of 3D stylized scenes, offering an effective and efficient solution for real-time 3D scene stylization.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Generative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
School of Humanities and Communication, Zhejiang Gongshang University, Hangzhou 310018, China
Funding
Zhejiang Province Program · Zhejiang Provincial Cultural Relics Protection Science and Technology Project · National Program of China · Macau project: Key technology research and display system development for new personalized controllable dressing dynamic display, and Ningbo Science and Technology Plan Project
References (32)
Show source-provided references
1. Gatys, L.A., Ecker, A.S., and Bethge, M. (2016, January 27–30). Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.265 ↗
2. Johnson, J., Alahi, A., and Li, F.-F. (2016, January 11–14). Perceptual losses for real-time style transfer and super-resolution. Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46475-6_43 ↗
3. Park, D.Y., and Lee, K.H. (2019, January 15–20). Arbitrary Style Transfer With Style-Attentional Networks. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00603 ↗
4. Chen, J., Xing, W., Sun, J., Chu, T., Huang, Y., Ji, B., Zhao, L., Lin, H., Chen, H., and Wang, Z. (2024, January 26–27). PNeSM: Arbitrary 3D scene stylization via prompt-based neural style mapping. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i2.27870 ↗
5. Nerf: Representing scenes as neural radiance fields for view synthesis10.1145/3503250 ↗
6. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). Imagenet classification with deep convolutional neural networks. Proceedings of the 26th Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
7. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
8. Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., and Yang, M.H. (2017, January 4–9). Universal style transfer via feature transforms. Proceedings of the 30th Advances in Neural Information Processing Systems, Long Beach, CA, USA.
9. Huang, X., and Belongie, S. (2017, January 22–29). Arbitrary style transfer in real-time with adaptive instance normalization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.167 ↗
10. Zhang, Y., Huang, N., Tang, F., Huang, H., Ma, C., Dong, W., and Xu, C. (2023, January 18–22). Inversion-based style transfer with diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00978 ↗
11. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual.
12. Wang, Z., Zhao, L., and Xing, W. (2023, January 1–6). Stylediffusion: Controllable disentangled style transfer via diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00706 ↗
13. Huang, H.P., Tseng, H.Y., Saini, S., Singh, M., and Yang, M.H. (2021, January 11–17). Learning to stylize novel views. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01361 ↗
14. Meshxl: Neural coordinate field for generative 3d foundation models
15. Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., and Hariharan, B. (November, January 27). Pointflow: 3D point cloud generation with continuous normalizing flows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
16. Liu, K., Zhan, F., Chen, Y., Zhang, J., Yu, Y., El Saddik, A., Lu, S., and Xing, E.P. (2023, January 17–24). Stylerf: Zero-shot 3D style transfer of neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00806 ↗
17. Zhang, Y., He, Z., Xing, J., Yao, X., and Jia, J. (2023, January 17–24). Ref-npr: Reference-based non-photorealistic radiance fields for controllable scene stylization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00413 ↗
18. Liu, K., Zhan, F., Xu, M., Theobalt, C., Shao, L., and Lu, S. (2024, January 3–6). Stylegaussian: Instant 3D style transfer with gaussian splatting. Proceedings of the SIGGRAPH Asia 2024 Technical Communications, Tokyo, Japan.10.1145/3681758.3698002 ↗
19. Galerne, B., Wang, J., Raad, L., and Morel, J.M. (2025, January 11–15). SGSST: Scaling Gaussian Splatting Style Transfer. Proceedings of the Computer Vision and Pattern Recognition Conference, Nashville, TN, USA.10.1109/CVPR52734.2025.02471 ↗
20. Jain, S., Kuthiala, A., Sethi, P.S., and Saxena, P. (2024). Stylesplat: 3d object style transfer with gaussian splatting. arXiv.
21. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022, January 19–20). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01042 ↗
22. Xiong, H. (2024). SparseGS: Real-Time 360° Sparse View Synthesis Using Gaussian Splatting, University of California.
23. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
24. Poole, B., Jain, A., Barron, J.T., and Mildenhall, B. (2022). Dreamfusion: Text-to-3D using 2D diffusion. arXiv.
25. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines10.1145/3306346.3322980 ↗
26. Tanks and temples: Benchmarking large-scale scene reconstruction10.1145/3072959.3073599 ↗
27. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., and Hedman, P. (2022, January 18–24). Mip-nerf 360: Unbounded anti-aliased neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00539 ↗
28. Schönberger, J.L., Zheng, E., Frahm, J.M., and Pollefeys, M. (2016, January 11–14). Pixelwise view selection for unstructured multi-view stereo. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part III 14.10.1007/978-3-319-46487-9_31 ↗
29. Cheng, K., Long, X., Yang, K., Yao, Y., Yin, W., Ma, Y., Wang, W., and Chen, X. (2024, January 21–27). Gaussianpro: 3D gaussian splatting with progressive propagation. Proceedings of the Forty-First International Conference on Machine Learning, Vienna, Austria.
30. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
31. Zhang, K., Kolkin, N., Bi, S., Luan, F., Xu, Z., Shechtman, E., and Snavely, N. (2022, January 23–27). Arf: Artistic radiance fields. Proceedings of the 17th European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19821-2_41 ↗
32. Huang, Y.H., He, Y., Yuan, Y.J., Lai, Y.K., and Gao, L. (2022, January 18–24). Stylizednerf: Consistent 3D scene stylization as stylized nerf via 2D-3D mutual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01780 ↗