BERTIVITS: The Posterior Encoder Fusion of Pre-Trained Models and Residual Skip Connections for End-to-End Speech Synthesis
Source abstract
Abstract
Enhancing the naturalness and rhythmicity of generated audio in end-to-end speech synthesis is crucial. The current state-of-the-art (SOTA) model, VITS, utilizes a conditional variational autoencoder architecture. However, it faces challenges, such as limited robustness, due to training solely on text and spectrum data from the training set. Particularly, the posterior encoder struggles with mid- and high-frequency feature extraction, impacting waveform reconstruction. Existing efforts mainly focus on prior encoder enhancements or alignment algorithms, neglecting improvements to spectrum feature extraction. In response, we propose BERTIVITS, a novel model integrating BERT into VITS. Our model features a redesigned posterior encoder with residual connections and utilizes pre-trained models to enhance spectrum feature extraction. Compared to VITS, BERTIVITS shows significant subjective MOS score improvements (0.16 in English, 0.36 in Chinese) and objective Mel-Cepstral coefficient reductions (0.52 in English, 0.49 in Chinese). BERTIVITS is tailored for single-speaker scenarios, improving speech synthesis technology for applications like post-class tutoring or telephone customer service.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China
References (38)
Show source-provided references
1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2023). Attention Is All You Need. arXiv.
2. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
3. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
4. Hsu, W.-N., Bolte, B., Tsai, Y.-H.H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. arXiv.10.1109/TASLP.2021.3122291 ↗
5. Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. arXiv.
6. Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. (2015, January 19–24). Librispeech: An ASR corpus based on public domain audio books. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, QLD, Australia.10.1109/ICASSP.2015.7178964 ↗
7. Kim, J., Kong, J., and Son, J. (2021, January 18–24). Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech. Proceedings of the 38th International Conference on Machine Learning, Virtual.
8. Nguyen, L.T., Pham, T., and Nguyen, D.Q. (2023). XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech. arXiv.
9. Zhang, J.-H., Zhang, Z.-H., Yan, Q., and Wang, P.-W. (2023). Speech emotion recognition based on energy frames and time frequency fusion. Comput. Sci., 1–11.
10. IEMOCAP: Interactive emotional dyadic motion capture database10.1007/s10579-008-9076-6 ↗
11. Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., and Bengio, S. (2017). Tacotron: Towards End-to-End Speech Synthesis. arXiv.10.21437/Interspeech.2017-1452 ↗
12. Expressive Speech Synthesis Method Based on Tacotron Model and Prosodic Correction
13. Speech Emotion Recognition Based on Improved MFCC and Parallel Hybrid Model
14. Livingstone, S.R., and Russo, F.A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13.10.1371/journal.pone.0196391 ↗
15. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W., and Weiss, B. (2005, January 4–8). A Database of German Emotional Speech. Proceedings of the Interspeech 2005, Lissabon, Portugal.10.21437/Interspeech.2005-446 ↗
16. Research on Teacher Speech Emotion Recognition in International Chinese Language Classrooms
17. Liao, S., Lan, S., and Zachariah, A.G. (2024). EVA-GAN: Enhanced Various Audio Generation via Scalable Generative Adversarial Networks. arXiv.
18. Kong, J., Park, J., Kim, B., Kim, J., Kong, D., and Kim, S. (2023). VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design. arXiv.10.21437/Interspeech.2023-534 ↗
19. Lei, Y., Yang, S., Cong, J., Xie, L., and Su, D. (2022). Glow-WaveGAN 2: High-quality Zero-shot Text-to-speech Synthesis and Any-to-any Voice Conversion. arXiv.10.21437/Interspeech.2022-684 ↗
20. Shen, J., Pang, R., Weiss, R.J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., and Skerrv-Ryan, R. (2018). Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. arXiv.10.1109/ICASSP.2018.8461368 ↗
21. Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). WaveNet: A Generative Model for Raw Audio. arXiv.
22. Im, C.-B., Lee, S.-H., Kim, S.-B., and Lee, S.-W. (2022, January 23–27). EMOQ-TTS: Emotion Intensity Quantization for Fine-Grained Controllable Emotional Text-to-Speech. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747098 ↗
23. Kim, M., Jeong, M., Choi, B.J., Kim, S., Lee, J.Y., and Kim, N.S. (2024). Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Prediction. arXiv.10.1109/ASRU57964.2023.10389791 ↗
24. Ju, Y., Kim, I., Yang, H., Kim, J.-H., Kim, B., Maiti, S., and Watanabe, S. (2022, January 18–22). TriniTTS: Pitch-controllable End-to-end TTS without External Aligner. Proceedings of the Interspeech 2022, ISCA, Incheon, Republic of Korea.10.21437/Interspeech.2022-925 ↗
25. Zhang, H., Yu, X., and Lin, Y. (2023, January 4–10). NSV-TTS: Non-Speech Vocalization Modeling and Transfer in Emotional Text-to-Speech. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096033 ↗
26. Liu, Z., Tian, Q., Hu, C., Liu, X., Wu, M., Wang, Y., Zhao, H., and Wang, Y. (2022). Controllable and Lossless Non-Autoregressive End-to-End Text-to-Speech. arXiv.
27. Tan, X., Chen, J., Liu, H., Cong, J., Zhang, C., Liu, Y., Wang, X., Leng, Y., Yi, Y., and He, L. (2022). NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality. arXiv.
28. Kingma, D.P., and Welling, M. (2022). Auto-Encoding Variational Bayes. arXiv.
29. Shirahata, Y., Yamamoto, R., Song, E., Terashima, R., Kim, J.-M., and Tachibana, K. (2023, January 4–10). Period VITS: Variational Inference with Explicit Pitch Modeling for End-To-End Emotional Speech Synthesis. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096480 ↗
30. Lin, T.-Q., Lee, H., and Tang, H. (2023). MelHuBERT: A simplified HuBERT on Mel spectrograms. arXiv.10.1109/ASRU57964.2023.10389700 ↗
31. Lam, M.W.Y., Wang, J., Su, D., and Yu, D. (2022). BDDM: Bilateral denoising diffusion models for fast and high-quality speech synthesis. arXiv.
32. Sohn, K., Lee, H., and Yan, X. (2015). Learning Structured Output Representation using Deep Conditional Generative Models. Advances in Neural Information Processing Systems, Curran Associates, Inc.. Available online: https://papers.nips.cc/paper_files/paper/2015/hash/8d55a249e6baa5c06772297520da2051-Abstract.html.
33. Kim, M., Jeong, M., Choi, B.J., Ahn, S., Lee, J.Y., and Kim, N.S. (2022, January 18–22). Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus. Proceedings of the Interspeech 2022, Incheon, Republic of Korea.10.21437/Interspeech.2022-225 ↗
34. (2024, May 14). Ljspeech: Ito, K, The LJ Speech Dataset. Available online: https://keithito.com/LJ-Speech-Dataset/.
35. (2024, May 14). Databaker: Databaker Technology, Chinese Standard Female Voice Database. Available online: https://www.data-baker.com/data/index/TNtts/.
36. Loshchilov, I., and Hutter, F. (2019). Decoupled Weight Decay Regularization. arXiv.
37. A Stochastic Approximation Method10.1214/aoms/1177729586 ↗
38. Saeki, T., Tachibana, K., and Yamamoto, R. (2022). DRSpeech: Degradation-Robust Text-to-Speech Synthesis with Frame-Level and Utterance-Level Acoustic Representation Learning. arXiv.10.21437/Interspeech.2022-294 ↗