Generative AI for Video Translation: A Scalable Architecture for Multilingual Video Conferencing
Source abstract
Abstract
The real-time deployment of cascaded generative AI pipelines for applications like video translation is constrained by significant system-level challenges. These include the cumulative latency of sequential model inference and the quadratic (O(N2)) computational complexity that renders multi-user video conferencing applications unscalable. This paper proposes and evaluates a practical system-level framework designed to mitigate these critical bottlenecks. The proposed architecture incorporates a turn-taking mechanism to reduce computational complexity from quadratic to linear in multi-user scenarios, and a segmented processing protocol to manage inference latency for a perceptually real-time experience. We implement a proof-of-concept pipeline and conduct a rigorous performance analysis across a multi-tiered hardware setup, including commodity (NVIDIA RTX 4060), cloud (NVIDIA T4), and enterprise (NVIDIA A100) GPUs. Our objective evaluation demonstrates that the system achieves real-time throughput (τ<1.0) on modern hardware. A subjective user study further validates the approach, showing that a predictable, initial processing delay is highly acceptable to users in exchange for a smooth, uninterrupted playback experience. The work presents a validated, end-to-end system design that offers a practical roadmap for deploying scalable, real-time generative AI applications in multilingual communication platforms.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Generative Adversarial Networks and Image Synthesis · Multimodal Machine Learning Applications · Embedded Systems Design Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Yildiz Technical University, Istanbul 34220, Turkey
R&D Center, Aktif Bank, Istanbul 34394, Turkey
References (54)
Show source-provided references
1. Doropoulos, S., Karapalidou, E., Charitidis, P., Karakeva, S., and Vologiannidis, S. (2025). Beyond manual media coding: Evaluating large language models and agents for news content analysis. Appl. Sci., 15.10.3390/app15148059 ↗
2. Oskooei, A.R., Babacan, M.S., Yağcı, E., Alptekin, Ç., and Buğday, A. (2024, January 19–21). Beyond Synthetic Benchmarks: Assessing Recent LLMs for Code Generation. Proceedings of the 14th International Workshop on Computer Science and Engineering (WCSE 2024), Phuket Island, Thailand.
3. Acosta-Bermejo, R., Terrazas-Chavez, J.A., and Aguirre-Anaya, E. (2025). Automated Malware Source Code Generation via Uncensored LLMs and Adversarial Evasion of Censored Model. Appl. Sci., 15.10.3390/app15179252 ↗
4. Liu, M., Wang, J., Lin, T., Ma, Q., Fang, Z., and Wu, Y. (2024). An empirical study of the code generation of safety-critical software using llms. Appl. Sci., 14.10.3390/app14031046 ↗
5. Kim, S., and Lee, K. (2024). DanceCaps: Pseudo-Captioning for Dance Videos Using Large Language Models. Appl. Sci., 14.10.3390/app142210116 ↗
6. Yan, J., Chen, Z., Cai, J., Xian, W., Wei, X., Qin, Y., and Li, Y. (2025). Video-Driven Artificial Intelligence for Predictive Modelling of Antimicrobial Peptide Generation: Literature Review on Advances and Challenges. Appl. Sci., 15.10.3390/app15137363 ↗
7. Oskooei, A.R., Caglar, E., Yakut, S., Tuten, Y.T., and Aktas, M.S. (July, January 30). Facial Stress and Fatigue Recognition via Emotion Weighting: A Deep Learning Approach. Proceedings of the International Conference on Computational Science and Its Applications, Istanbul, Türkiye.
8. Partarakis, N., Evdaimon, T., Katsantonis, M., and Zabulis, X. (2025). Training First Responders Through VR-Based Situated Digital Twins. Computers, 14.10.3390/computers14070274 ↗
9. Todino, M.D., Pitri, E., Fella, A., Michaelidou, A., Campitiello, L., Placanica, F., Di Tore, S., and Sibilio, M. (2025). Bridging Tradition and Innovation: Transformative Educational Practices in Museums with AI and VR. Computers, 14.10.3390/computers14070257 ↗
10. Johnson, D., Mamani, B., and Salas, C. (2024). Collabvr: VR testing for increasing social interaction between college students. Computers, 13.10.3390/computers13020040 ↗
11. Jover, V., and Sempere, S. (2025). Creating Digital Twins to Celebrate Commemorative Events in the Metaverse. Computers, 14.10.3390/computers14070273 ↗
12. Mohamed, A., Faisal, R., Al-Gindy, A., and Shaalan, K. (2025). Artificial Intelligence and Immersive Technologies: Virtual Assistants in AR/VR for Special Needs Learners. Computers, 14.10.3390/computers14080306 ↗
13. Jun, S. (2025). Patent Keyword Analysis Using Bayesian Factor Analysis and Social Network Visualization in Digital Therapy Technology. Computers, 14.10.3390/computers14030078 ↗
14. Huitema, I.R., Oskooei, A.R., Aktaş, M.S., and Riveni, M. (2024, January 10–12). Investigating Echo-Chambers in Decentralized Social Networks: A Mastodon Case Study. Proceedings of the International Conference on Complex Networks and Their Applications, Istanbul, Turkey.10.1007/978-3-031-82431-9_26 ↗
15. Wang, H., Wang, C., and Qi, H. (2025). Research on the Network Structure Characteristics of Doctors and the Influencing Mechanism on Recommendation Rates in Online Health Communities: A Multi-Dimensional Perspective Based on the “Good Doctor Online” Platform. Appl. Sci., 15.10.3390/app15084583 ↗
16. Soares, N.F., Marin, A., Rodrigues, D.L., and Duarte, F.J. (July, January 30). Digital Twin and Metaverse in the Context of Industry 4.0. Proceedings of the International Conference on Computational Science and Its Applications, Istanbul, Türkiye.
17. Leung, C., and Xu, Z. (2024, January 1–4). Large Language Models for Emotion Evolution Prediction. Proceedings of the International Conference on Computational Science and Its Applications, Hanoi, Vietnam.10.1007/978-3-031-65154-0_1 ↗
18. Rafiei Oskooei, A., Caglar, E., Şahin, I., Kayabay, A., and Aktas, M.S. (July, January 30). Whisper, Translate, Speak, Sync: Video Translation for Multilingual Video Conferencing Using Generative AI. Proceedings of the International Conference on Computational Science and Its Applications, Istanbul, Türkiye.
19. Automatic speech recognition: A survey10.1007/s11042-020-10073-7 ↗
20. End-to-end speech recognition: A survey10.1109/TASLP.2023.3328283 ↗
21. Automatic speech recognition using advanced deep learning approaches: A survey10.1016/j.inffus.2024.102422 ↗
22. NLLB Team (2024). Scaling neural machine translation to 200 languages. Nature, 630, 841.10.1038/s41586-024-07335-x ↗
23. Progress in machine translation10.1016/j.eng.2021.03.023 ↗
24. Zhu, W., Liu, H., Dong, Q., Xu, J., Huang, S., Kong, L., Chen, J., and Li, L. (2023). Multilingual machine translation with large language models: Empirical results and analysis. arXiv.10.18653/v1/2024.findings-naacl.176 ↗
25. Oskooei, A.R., Yukcu, S., Bozoglan, M.C., and Aktas, M.S. (2025). Repository-Level Code Understanding by LLMs via Hierarchical Summarization: Improving Code Search and Bug Localization. International Conference on Computational Science and Its Applications, Springer.10.1007/978-3-031-97576-9_6 ↗
26. Casanova, E., Davis, K., Gölge, E., Göknar, G., Gulea, I., Hart, L., Aljafari, A., Meyer, J., Morais, R., and Olayemi, S. (2024). XTTS: A Massively Multilingual Zero-Shot Text-to-Speech Model. arXiv.10.21437/Interspeech.2024-2016 ↗
27. Li, R., Pu, D., Huang, M., and Huang, B. (2022, January 23–27). Unet-tts: Improving unseen speaker and style transfer in one-shot voice cloning. Proceedings of the ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746049 ↗
28. Chu, S.N., and Goodell, A.J. (2024). Synthetic Patients: Simulating Difficult Conversations with Multimodal Generative AI for Medical Education. arXiv.
29. KR, P., Mukhopadhyay, R., Philip, J., Jha, A., Namboodiri, V., and Jawahar, C. (2019, January 21–25). Towards automatic face-to-face translation. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.
30. Prajwal, K., Mukhopadhyay, R., Namboodiri, V.P., and Jawahar, C. (2020, January 12–16). A lip sync expert is all you need for speech to lip generation in the wild. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.10.1145/3394171.3413532 ↗
31. MILG: Realistic lip-sync video generation with audio-modulated image inpainting10.1016/j.visinf.2024.08.002 ↗
32. Lip to speech synthesis with visual context attentional gan
33. Vougioukas, K., Petridis, S., and Pantic, M. (2019, January 16–20). End-to-End Speech-Driven Realistic Facial Animation with Temporal GANs. Proceedings of the CVPR Workshops, Long Beach, CA, USA.
34. Das, D., Biswas, S., Sinha, S., and Bhowmick, B. (2020, January 23–28). Speech-driven facial animation using cascaded gans for learning of motion and texture. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XXX 16.10.1007/978-3-030-58577-8_25 ↗
35. Yin, F., Zhang, Y., Cun, X., Cao, M., Fan, Y., Wang, X., Bai, Q., Wu, B., Wang, J., and Yang, Y. (2022, January 23–27). Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19790-1_6 ↗
36. Hong, F.T., Zhang, L., Shen, L., and Xu, D. (2022, January 18–24). Depth-aware generative adversarial network for talking head video generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00339 ↗
37. Lin, G., Jiang, J., Yang, J., Zheng, Z., and Liang, C. (2025). OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models. arXiv.
38. Li, C., Zhang, C., Xu, W., Xie, J., Feng, W., Peng, B., and Xing, W. (2024). LatentSync: Audio Conditioned Latent Diffusion Models for Lip Sync. arXiv.
39. Xu, S., Chen, G., Guo, Y.X., Yang, J., Li, C., Zang, Z., Zhang, Y., Tong, X., and Guo, B. (2024). Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv.10.52202/079017-0021 ↗
40. Shen, S., Zhao, W., Meng, Z., Li, W., Zhu, Z., Zhou, J., and Lu, J. (2023, January 18–22). Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00197 ↗
41. Stypułkowski, M., Vougioukas, K., He, S., Zięba, M., Petridis, S., and Pantic, M. (2024, January 3–8). Diffused heads: Diffusion models beat gans on talking-face generation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV57701.2024.00502 ↗
42. Ye, Z., He, J., Jiang, Z., Huang, R., Huang, J., Liu, J., Ren, Y., Yin, X., Ma, Z., and Zhao, Z. (2023). Geneface++: Generalized and stable real-time audio-driven 3d talking face generation. arXiv.
43. Sun, X., Zhang, L., Zhu, H., Zhang, P., Zhang, B., Ji, X., Zhou, K., Gao, D., Bo, L., and Cao, X. (2023). Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior. arXiv.
44. Ye, Z., Zhong, T., Ren, Y., Jiang, Z., Huang, J., Huang, R., Liu, J., He, J., Zhang, C., and Wang, Z. (2024). MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes. arXiv.
45. Ye, Z., Zhong, T., Ren, Y., Yang, J., Li, W., Huang, J., Jiang, Z., He, J., Huang, R., and Liu, J. (2024). Real3d-portrait: One-shot realistic 3d talking portrait synthesis. arXiv.
46. Wav2NeRF: Audio-driven realistic talking head generation via wavelet-based NeRF10.1016/j.imavis.2024.105104 ↗
47. Song, W., Liu, Q., Liu, Y., Zhang, P., and Cao, J. (2025). Multi-Level Feature Dynamic Fusion Neural Radiance Fields for Audio-Driven Talking Head Generation. Appl. Sci., 15.10.3390/app15010479 ↗
48. Zhao, Z., Zhang, Y., Wu, T., Guo, H., and Li, Y. (2022). Emotionally controllable talking face generation from an arbitrary emotional portrait. Appl. Sci., 12.10.3390/app122412852 ↗
49. Naebi, A., and Feng, Z. (2023). The Performance of a Lip-Sync Imagery Model, New Combinations of Signals, a Supplemental Bond Graph Classifier, and Deep Formula Detection as an Extraction and Root Classifier for Electroencephalograms and Brain–Computer Interfaces. Appl. Sci., 13.10.20944/preprints202308.0528.v1 ↗
50. Rafiei Oskooei, A., Yahsi, E., Sungur, M., and Aktas, M.S. (2024, January 1–4). Can One Model Fit All? An Exploration of Wav2Lip’s Lip-Syncing Generalizability Across Culturally Distinct Languages. Proceedings of the International Conference on Computational Science and Its Applications, Hanoi, Vietnam.10.1007/978-3-031-65282-0_10 ↗
51. Ritter, M., Meier, U., Yang, J., and Waibel, A. (1999, January 7–10). Face translation: A multimodal translation agent. Proceedings of the AVSP’99-International Conference on Auditory-Visual Speech Processing, Santa Cruz, CA, USA.
52. Song, H.K., Woo, S.H., Lee, J., Yang, S., Cho, H., Lee, Y., Choi, D., and Kim, K.W. (2022, January 18–24). Talking face generation with multilingual tts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.02074 ↗
53. Cheng, X., Huang, R., Li, L., Jin, T., Wang, Z., Yin, A., Li, M., Duan, X., Huai, B., and Zhao, Z. (2023). TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation. arXiv.10.18653/v1/2024.findings-acl.593 ↗
54. Rafiei Oskooei, A., Aktaş, M.S., and Keleş, M. (2024). Seeing the Sound: Multilingual Lip Sync for Real-Time Face-to-Face Translation. Computers, 14.10.3390/computers14010007 ↗