Tibetan Speech Synthesis Based on Pre-Traind Mixture Alignment FastSpeech2
Source abstract
Abstract
Most current research in Tibetan speech synthesis relies primarily on autoregressive models in deep learning. However, these models face challenges such as slow inference, skipped readings, and repetitions. To overcome these issues, we propose an enhanced non-autoregressive acoustic model combined with a vocoder for Tibetan speech synthesis. Specifically, we introduce the mixture alignment FastSpeech2 method to correct errors caused by hard alignment in the original FastSpeech2 method. This new method employs soft alignment at the level of Latin letters and hard alignment at the level of Tibetan characters, thereby improving alignment accuracy between text and speech and enhancing the naturalness and intelligibility of the synthesized speech. Additionally, we integrate pitch and energy information into the model, further enhancing overall synthesis quality. Furthermore, Tibetan has relatively smaller text-to-audio datasets compared to widely studied languages. To address these limited resources, we employ a transfer learning approach to pre-train the model with data from resource-rich languages. Subsequently, this pre-trained mixture alignment FastSpeech2 model is fine-tuned for Tibetan speech synthesis. Experimental results demonstrate that the mixture alignment FastSpeech2 model produces higher-quality speech compared to the original FastSpeech2 model, particularly when pre-trained on an English dataset, resulting in further improvements in clarity and naturalness.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Music and Audio Processing · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China, Beijing 100081, China
School of Information Engineering, Minzu University of China, Beijing 100081, China
Funding
National Natural Science Foundation of China
References (24)
Show source-provided references
1. Current Status and Development Trends of Voice Interaction Technology on Mobile Intelligent Terminals
2. Jia, Y., Zhang, Y., Weiss, R., Wang, Q., Shen, J., Ren, F., Nguyen, P., Pang, R., Lopez Moreno, I., and Wu, Y. (2018, January 3–8). Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
3. Arık, S.Ö., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., and Raiman, J. (2017, January 6–11). Deep voice: Real-time neural text-to-speech. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
4. Sotelo, J., Mehri, S., Kumar, K., Santos, J.F., Kastner, K., Courville, A., and Bengio, Y. (2017). Char2wav: End-to-End Speech Synthesis, International Speech Communication Association.
5. Donahue, C., McAuley, J., and Puckette, M. (2018). Adversarial audio synthesis. arXiv.
6. Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). Wavenet: A generative model for raw audio. arXiv.
7. Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., and Bengio, S. (2017). Tacotron: Towards end-to-end speech synthesis. arXiv.10.21437/Interspeech.2017-1452 ↗
8. Shen, J., Pang, R., Weiss, R.J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., and Skerrv-Ryan, R. (2018, January 15–20). Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461368 ↗
9. Skerry-Ryan, R., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R., Clark, R., and Saurous, R.A. (2018, January 10–15). Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. Proceedings of the 35th International Conference on Machine Learning (ICML 2018), Stockholm, Sweden.
10. Li, N., Liu, S., Liu, Y., Zhao, S., and Liu, M. (February, January 27). Neural speech synthesis with transformer network. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Honolulu, HI, USA.
11. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.Y. (2019, January 8–14). Fastspeech: Fast, robust and controllable text to speech. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
12. Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.Y. (2020). Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv.
13. Research on HMM-based Tibetan speech synthesis
14. Xie, X. (2021). Research on the Speech Synthesis Technology of Tibetan Dialect in Lhasa-Ü-Tsang. [Master’s Thesis, Department of Information Science and Technology, Tibet University].
15. Signal estimation from modified short-time Fourier transform10.1109/TASSP.1984.1164317 ↗
16. Lhasa-Tibetan speech synthesis using end-to-end model10.1109/ACCESS.2019.2940125 ↗
17. Tian, C., Ma, J., Zhang, C., and Zhan, P. (2018). A deep neural network model for short-term load forecast based on long short-term memory network and convolutional neural network. Energies, 11.10.3390/en11123493 ↗
18. Xu, X., Li, N., and Zhao, Y. (2023, January 18–20). Transformer Speech Synthesis for Tibetan Lhasa Dialect Based on Multilevel Granular Unit. Proceedings of the 2023 IEEE 6th International Conference on Pattern Recognition and Artificial Intelligence (PRAI), Haikou, China.10.1109/PRAI59366.2023.10331979 ↗
19. Zu, B., Cai, R., Cai, Z., and Pengmao, Z. (2022, January 22–24). Research on Tibetan Speech Synthesis Based on Fastspeech2. Proceedings of the 2022 3rd International Conference on Pattern Recognition and Machine Learning (PRML), Chengdu, China.10.1109/PRML56267.2022.9882187 ↗
20. McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., and Sonderegger, M. (2017, January 20–24). Montreal forced aligner: Trainable text-speech alignment using kaldi. Proceedings of the Interspeech, Stockholm, Sweden.10.21437/Interspeech.2017-1386 ↗
21. Ren, Y., Liu, J., and Zhao, Z. (2021, January 6–14). Portaspeech: Portable and high-quality generative text-to-speech. Proceedings of the 35th International Conference on Neural Information Processing Systems, Online.
22. Vietnamese speech synthesis based on transfer learning
23. Kong, J., Kim, J., and Bae, J. (2020, January 6–12). Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
24. Ito, K., and Johnson, L. (2024, July 29). The LJSpeech Dataset. Available online: https://keithito.com/LJ-Speech-Dataset/.