Meta-Learning for Mandarin-Tibetan Cross-Lingual Speech Synthesis
Source abstract
Abstract
The paper proposes a meta-learning-based Mandarin-Tibetan cross-lingual text-to-speech (TTS) to realize both Mandarin and Tibetan speech synthesis under a unique framework. First, we build two kinds of Tacotron2-based Mandarin-Tibetan cross-lingual baseline TTS. One is a shared encoder Mandarin-Tibetan cross-lingual TTS, and another is a separate encoder Mandarin-Tibetan cross-lingual TTS. Both baseline TTS use the speaker classifier with a gradient reversal layer to disentangle speaker-specific information from the text encoder. At the same time, we design a prosody generator to extract prosodic information from sentences to explore syntactic and semantic information adequately. To further improve the synthesized speech quality of the Tacotron2-based Mandarin-Tibetan cross-lingual TTS, we propose a meta-learning-based Mandarin-Tibetan cross-lingual TTS. Based on the separate encoder Mandarin-Tibetan cross-lingual TTS, we use an additional dynamic network to predict the parameters of the language-dependent text encoder that could realize better cross-lingual knowledge sharing in the sequence-to-sequence TTS. Lastly, we synthesize Mandarin or Tibetan speech through the unique acoustic model. The baseline experimental results show that the separate encoder Mandarin-Tibetan cross-lingual TTS could handle the input of different languages better than the shared encoder Mandarin-Tibetan cross-lingual TTS. The experimental results further show that the proposed meta-learning-based Mandarin-Tibetan cross-lingual speech synthesis method could effectively improve the voice quality of synthesized speech in terms of naturalness and speaker similarity.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Natural Language Processing Techniques · Speech and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Physics and Electronic Engineering, Northwest Normal University, Lanzhou 730070, China
School of Educational Technology, Northwest Normal University, Lanzhou 730070, China
National and Provincial Joint Engineering Laboratory of Learning Analysis Technology in Online Education, Lanzhou 730070, China
Funding
Science and Technology Program of Gansu Province · University Innovation Foundation of Gansu Province · National Natural Science Foundation of China
References (49)
Show source-provided references
1. Neural network based autonomous control of a speech synthesis system
2. Hunt, A.J., and Black, A.W. (1996, January 9). Unit selection in a concatenative speech synthesis system using a large speech database. Proceedings of the 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing, Atlanta, GA, USA.
3. Speech synthesis based on hidden markov models10.1109/JPROC.2013.2251852 ↗
4. Ze, H., Senior, A., and Schuster, M. (2013, January 26–31). Statistical parametric speech synthesis using deep neural networks. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech, and Signal Processing, Vancouver, BC, Canada.
5. Wang, Y., Skerry-Ryan, R.J., Stanton, D., Wu, Y., and Saurous, R.A. (2017, January 20–24). Tacotron: Towards end-to-end speech synthesis. Proceedings of the 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden.10.21437/Interspeech.2017-1452 ↗
6. Li, N., Liu, S., Liu, Y., Zhao, S., and Liu, M. (February, January 27). Neural speech synthesis with Transformer network. Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.10.1609/aaai.v33i01.33016706 ↗
7. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., and Zhao, Z. (2019, January 8–14). Fastspeech: Fast, robust and controllable text to speech. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
8. Tachibana, M., Izawa, S., Nose, T., and Kobayashi, T. (April, January 31). Speaker and style adaptation using average voice model for style control in hmm-based speech synthesis. Proceedings of the 2008 IEEE International Conference on Acoustics, Speech, and Signal Processing, Las Vegas, NV, USA.
9. Speaker interpolation in hmm-based speech synthesis system
10. Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends10.1109/MSP.2014.2359987 ↗
11. Modeling spectral envelopes using restricted boltzmann machines and deep belief networks for statistical parametric speech synthesis
12. Kang, S., Qian, X., and Meng, H. (2013, January 26–31). Multi-distribution deep belief network for speech synthesis. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech, and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6639225 ↗
13. Zen, H., and Senior, A.W. (2014, January 4–9). Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech, and Signal Processing, Florence, Italy.10.1109/ICASSP.2014.6854321 ↗
14. Effective spectral and excitation modeling techniques for LSTM-RNN-based speech synthesis systems
15. Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7—9). Neural machine translation by jointly learning to align and translate. Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA.
16. Luong, M.T., Pham, H., and Manning, C.D. (2015, January 17–21). Effective approaches to attention-based neural machine translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1166 ↗
17. Chorowski, J., Bahdanau, D., Cho, K., and Bengio, Y. (2014, January 8–13). End-to-end continuous speech recognition using attention-based recurrent nn: First results. Proceedings of the 28th Conference on Neural Information Processing Systems Workshop on Deep Learning and Representation Learning, Montreal, QC, Canada.
18. Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., and Bengio, Y. (2015, January 7–12). Attention-based models for speech recognition. Proceedings of the 29th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
19. Sotelo, J., Mehri, S., Kumar, K., Santos, J.F., Kastner, K., and Courville, A.C. (2017, January 24–26). Char2Wav: End-to-end speech synthesis. Proceedings of the 5th International Conference on Learning Representations, Toulon, France.
20. Mehri, S., Kumar, K., Gulrajani, I., Kumar, R., Jain, S., and Sotelo, J. (2017, January 24–26). SampleRNN: An unconditional end-to-end neural audio generation model. Proceedings of the 5th International Conference on Learning Representations, Toulon, France.
21. Shen, J., Pang, R., Weiss, R.J., Schuster, M., Jaitly, N., and Yang, Z. (2018, January 15–20). Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada.10.1109/ICASSP.2018.8461368 ↗
22. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., and Gomez, A.N. (2017, January 4–9). Attention is all you need. Proceedings of the the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
23. Traber, C., Huber, K., Nedir, K., Pfister, B., Keller, E., and Zellner, B. (1999, January 5–9). From multilingual to polyglot speech synthesis. Proceedings of the 6th European Conference on Speech Communication and Technology, Budapest, Hungary.10.21437/Eurospeech.1999-203 ↗
24. A cross-language state sharing and mapping approach to bilingual (Mandarin–English) TTS10.1109/TASL.2009.2015708 ↗
25. Statistical parametric speech synthesis based on speaker and language factorization10.1109/TASL.2012.2187195 ↗
26. Nachmani, E., and Wolf, L. (2019, January 12–17). Unsupervised polyglot text-to-speech. Proceedings of the 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, Brighton, UK.10.1109/ICASSP.2019.8683519 ↗
27. Zhang, Y., Weiss, R.J., Zen, H., Wu, Y., Chen, Z., and Skerry-Ryan, R. (2019, January 15–19). Learning to speak fluently in a foreign language: Multilingual speech synthesis and cross-language voice cloning. Proceedings of the 20th Annual Conference of the International Speech Communication Association, Graz, Austria.10.21437/Interspeech.2019-2668 ↗
28. Cao, Y., Liu, S., Wu, X., Kang, S., and Meng, H. (2020, January 4–8). Code-switched speech synthesis using bilingual phonetic posteriorgram with only monolingual corpora. Proceedings of the 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain.10.1109/ICASSP40776.2020.9053094 ↗
29. Ning, Y., Wu, Z., Li, R., Jia, J., and Cai, L. (2016, January 20–25). Learning cross-lingual information with multilingual BLSTM for speech synthesis of low-resource languages. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech, and Signal Processing, Shanghai, China.
30. Fan, Y. (2016, January 20–25). Speaker and language factorization in DNN-based TTS synthesis. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech, and Signal Processing, Shanghai, China.10.1109/ICASSP.2016.7472737 ↗
31. Cao, Y., Wu, X., Liu, S., Yu, J., and Meng, H.M. (2019, January 12–17). End-to-end code-switched TTS with mix of monolingual recordings. Proceedings of the 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, Brighton, UK.10.1109/ICASSP.2019.8682927 ↗
32. Li, B., Zhang, Y., Sainath, T., Wu, Y., and Chan, W. (2019, January 12–17). Bytes are all you need: End-to-end multilingual speech recognition and synthesis with bytes. Proceedings of the 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, Brighton, UK.10.1109/ICASSP.2019.8682674 ↗
33. Zhan, H., Zhang, H., Ou, W., and Lin, Y. (September, January 31). Improve cross-Lingual text-to-speech synthesis on monolingual corpora with pitch contour information. Proceedings of the 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia.
34. Meta-learning in neural networks: A Survey
35. Klein, B., Wolf, L., and Afek, Y. (2015, January 7–12). A dynamic convolutional layer for short rangeweather prediction. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299117 ↗
36. Jia, X., De Brabandere, B., Tuytelaars, T., and Gool, L.V. (2015, January 5–10). Dynamic filter networks. Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain.
37. Ha, D., Dai, A., and Le, Q.V. (2017, January 24–26). Hypernetworks. Proceedings of the 5th International Conference on Learning Representations, Toulon, France.
38. Nekvinda, T., and Dušek, O. (2020, January 25–29). One model, many languages: Meta-learning for multilingual text-to-speech. Proceedings of the 21st Annual Conference of the International Speech Communication Association, Shanghai, China.10.21437/Interspeech.2020-2679 ↗
39. Using speaker adaptive training to realize mandarin-tibetan cross-lingual speech synthesis
40. Deep learning for Mandarin-Tibetan cross-lingual speech synthesis10.1109/ACCESS.2019.2954342 ↗
41. Xu, S., Yu, H., and Li, G. (2017, January 25–26). The influence of context on Tibetan Lhasa speech synthesis. Proceedings of the IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference, Chongqing, China.10.1109/IAEAC.2017.8054091 ↗
42. Lhasa-tibetan speech synthesis using end-to-end model
43. End-to-end speech synthesis for tibetan multidialect
44. Lu, Y., Dong, M., and Chen, Y. (2019, January 12–17). Implementing prosodic phrasing in Chinese end-to-end speech synthesis. Proceedings of the 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, Brighton, UK.10.1109/ICASSP.2019.8682368 ↗
45. Li, J., Wu, Z., Li, R., Zhi, P., and Meng, H. (2019, January 15–19). Knowledge-based linguistic encoding for end-to-end Mandarin text-to-speech synthesis. Proceedings of the 20th Annual Conference of the International Speech Communication Association, Graz, Austria.10.21437/Interspeech.2019-1118 ↗
46. Hayashi, T., Watanabe, S., Toda, T., Takeda, K., and Livescu, K. (2019, January 15–19). Pre-trained text embeddings for enhanced text-to-speech synthesis. Proceedings of the 20th Annual Conference of the International Speech Communication Association, Graz, Austria.10.21437/Interspeech.2019-3177 ↗
47. Xiao, Y., He, L., Ming, H., and Soong, F.K. (2020, January 4–8). Improving prosody with linguistic and bert derived features in multi-speaker based Mandarin Chinese neural TTS. Proceedings of the 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain.10.1109/ICASSP40776.2020.9054337 ↗
48. Zhang, W., and Yang, Y. (ACM Trans. Asian Low-Reso., 2022). Improving sequence-to-sequence Tibetan speech synthesis with prosodic information, ACM Trans. Asian Low-Reso., submitted.10.1145/3616012 ↗
49. Tachibana, H., Uenoyama, K., and Aihara, S. (2018, January 15–20). Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, Calgary, AB, Canada.10.1109/ICASSP.2018.8461829 ↗