FastTalk: Speech-Driven Lip Synchronization Video Generation for Chinese-Language Scenarios
Source abstract
Abstract
Speech-driven lip synchronization is an important technique for talking-face video generation, with broad application potential in virtual humans, video dubbing, digital media, and human–computer interaction. However, existing methods still face challenges in achieving reliable lip synchronization while maintaining stable identity preservation, high visual fidelity, and efficient inference, especially in Chinese-language scenarios where related research remains relatively limited. To address these issues, we propose FastTalk, a speech-driven lip synchronization method for Chinese-language scenarios. The proposed framework performs latent-space restoration for efficient video synthesis, uses a fixed-mask strategy to suppress shortcut visual cues and strengthen audio-driven lip-shape prediction, and adopts a two-stage training scheme to reduce the gap between training and inference. This design improves generation stability while preserving efficiency. Experimental results show that FastTalk achieves competitive lip synchronization performance while improving visual quality and identity preservation. These results indicate that FastTalk provides an effective solution for Chinese speech-driven lip synchronization video generation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Face recognition and analysis · Generative Adversarial Networks and Image Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201600, China
References (33)
Show source-provided references
1. Chung, J.S., Senior, A., Vinyals, O., and Zisserman, A. (2017). Lip-reading sentences in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2017.367 ↗
2. Chen, L., Maddox, R.K., Duan, Z., and Xu, C. (2019). Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR.2019.00802 ↗
3. Prajwal, K.R., Mukhopadhyay, R., Namboodiri, V.P., and Jawahar, C.V. (2020). A lip sync expert is all you need for speech to lip generation in the wild. Proceedings of the 28th ACM International Conference on Multimedia, Association for Computing Machinery.10.1145/3394171.3413532 ↗
4. Guo, Y., Chen, K., Liang, S., Liu, Y.-J., Bao, H., and Zhang, J. (2021). AD-NeRF: Audio driven neural radiance fields for talking head synthesis. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, IEEE Computer Society.10.1109/ICCV48922.2021.00573 ↗
5. Deep video portraits
6. Grassal, P.-W., Prinzler, M., Leistner, T., Rother, C., Nießner, M., and Thies, J. (2022). Neural head avatars from monocular RGB videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE Computer Society.10.1109/CVPR52688.2022.01810 ↗
7. Wang, S., Li, L., Ding, Y., Fan, C., and Yu, X. (2021). Audio2Head: Audio-driven one-shot talking-head generation with natural head motion. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence.10.24963/ijcai.2021/152 ↗
8. Zhang, W., Cun, X., Wang, X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., and Wang, F. (2023). SadTalker: Learning realistic 3D motion coefficients for stylized audio-driven single image talking face animation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE Computer Society.10.1109/CVPR52729.2023.00836 ↗
9. First order motion model for image animation
10. MakeItTalk: Speaker-aware talking-head animation
11. Cheng, K., Cun, X., Zhang, Y., Xia, M., Yin, F., Zhu, M., Wang, X., Wang, J., and Wang, N. (2022). VideoReTalking: Audio-based lip synchronization for talking head video editing in the wild. Proceedings of SIGGRAPH Asia, Association for Computing Machinery.10.1145/3550469.3555399 ↗
12. Wang, K., Zhang, J., Huang, J., Li, Q., Sun, M.T., Sakai, K., and Ku, W.S. (2023). CA-Wav2Lip: Coordinate attention-based speech to lip synthesis in the wild. Proceedings of the 2023 IEEE International Conference on Smart Computing, IEEE.10.1109/SMARTCOMP58114.2023.00018 ↗
13. Yaman, D., Eyiokur, F.I., Bärmann, L., Ekenel, H.K., and Waibel, A. (2024). Audio-driven talking face generation with stabilized synchronization loss. Proceedings of the European Conference on Computer Vision, Springer Nature.10.1007/978-3-031-72655-2_24 ↗
14. Shen, S., Zhao, W., Meng, Z., Li, W., Zhu, Z., Zhou, J., and Lu, J. (2023). DiffTalk: Crafting diffusion models for generalized audio-driven portraits animation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE Computer Society.10.1109/CVPR52729.2023.00197 ↗
15. Aneja, S., Thies, J., Dai, A., and Nießner, M. (2024). FaceTalk: Audio-driven motion diffusion for neural parametric head models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE Computer Society.10.1109/CVPR52733.2024.02009 ↗
16. Peng, Z., Hu, W., Shi, Y., Zhu, X., Zhang, X., Zhao, H., He, J., Liu, H., and Fan, Z. (2024, January 17–21). SyncTalk: The devil is in the synchronization for talking head synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00070 ↗
17. Gupta, A., Mukhopadhyay, R., Balachandra, S., Khan, F.F., Namboodiri, V.P., and Jawahar, C.V. (2023). Towards generating ultra-high resolution talking-face videos with lip synchronization. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, IEEE Computer Society.10.1109/WACV56688.2023.00518 ↗
18. Lee, C.-Y., Oh, H.-B., Han, E., Kim, S.-B., Nam, S., and Oh, T.-H. (2025). Perceptually accurate 3D talking head generation: New definitions, speech-mesh representation, and evaluation metrics. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE Computer Society.
19. Ji, X., Hu, X., Xu, Z., Zhu, J., Lin, C., He, Q., Zhang, J., Luo, D., Chen, Y., and Lin, Q. (2025). Sonic: Shifting focus to global audio perception in portrait animation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE Computer Society.10.1109/CVPR52734.2025.00027 ↗
20. Ki, T., Min, D., and Chae, G. (2025). FLOAT: Generative motion latent flow matching for audio-driven talking portrait. Proceedings of the 2025 IEEE/CVF International Conference on Computer Vision, IEEE Computer Society.10.1109/ICCV51701.2025.01364 ↗
21. Agarwal, M., Zhang, M., Sevilla-Lara, L., and McDonagh, S. (2026). GaussianHeadTalk: Wobble-free 3D talking heads with audio driven Gaussian splatting. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, IEEE Computer Society.
22. Audio-driven single image talking face animation with transformers10.1038/s41598-025-33928-1 ↗
23. Audio-visual synchronization-based lip synthesis
24. Kim, M., Yeo, J.H., Choi, J., and Ro, Y.M. (2023, January 2–6). Lip reading for low-resource languages by learning and combining general speech knowledge and language-specific knowledge. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01409 ↗
25. Song, H.-K., Woo, S.H., Lee, J., Yang, S., Cho, H., Lee, Y., Choi, D., and Kim, K.-w. (2022, January 21–24). Talking face generation with multilingual TTS. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.02074 ↗
26. Zhang, Y., Yang, S., Shan, S., and Chen, X. (2024, January 17–21). ES3: Evolving self-supervised learning of robust audio-visual speech representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02556 ↗
27. Zhao, Y., Xu, R., and Song, M. (2019). A cascade sequence-to-sequence model for Chinese Mandarin lip reading. Proceedings of the 1st ACM International Conference on Multimedia in Asia, Association for Computing Machinery.10.1145/3338533.3366579 ↗
28. Zhao, Y., Xu, R., and Wang, X. (2020). Hearing lips: Improving lip reading by distilling speech recognizers. Proceedings of the AAAI Conference on Artificial Intelligence, AAAI Press.10.1609/aaai.v34i04.6174 ↗
29. GANs trained by a two time-scale update rule converge to a local Nash equilibrium
30. Deng, J., Guo, J., Xue, N., and Zafeiriou, S. (2019). ArcFace: Additive angular margin loss for deep face recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE Computer Society.10.1109/CVPR.2019.00482 ↗
31. Wang, Q., Wu, D., and Xu, Z. (2025). JoyGen: Audio-driven 3D depth-aware talking-face video editing. arXiv.
32. Zhong, W., Fang, C., Cai, Y., Wei, P., Zhao, G., Lin, L., and Li, G. (2023). Identity-preserving talking face generation with landmark and appearance priors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE Computer Society.10.1109/CVPR52729.2023.00938 ↗
33. Zhang, Y., Liu, M., and Chen, Z. (2024). MuseTalk: Real-time high quality lip synchronization with latent space inpainting. arXiv.