A Review of Deep Learning Based Speech Synthesis
Source abstract
Abstract
Speech synthesis, also known as text-to-speech (TTS), has attracted increasingly more attention. Recent advances on speech synthesis are overwhelmingly contributed by deep learning or even end-to-end techniques which have been utilized to enhance a wide range of application scenarios such as intelligent speech interaction, chatbot or conversational artificial intelligence (AI). For speech synthesis, deep learning based techniques can leverage a large scale of pairs to learn effective feature representations to bridge the gap between text and speech, thus better characterizing the properties of events. To better understand the research dynamics in the speech synthesis field, this paper firstly introduces the traditional speech synthesis methods and highlights the importance of the acoustic modeling from the composition of the statistical parametric speech synthesis (SPSS) system. It then gives an overview of the advances on deep learning based speech synthesis, including the end-to-end approaches which have achieved start-of-the-art performance in recent years. Finally, it discusses the problems of the deep learning methods for speech synthesis, and also points out some appealing research directions that can bring the speech synthesis research into a new frontier.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Music and Audio Processing · Speech and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Research Institute of Information Technology Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
Department of Computer Science and Technology Institute of Internet Industry, Tsinghua University, Beijing 100084, China
National Engineering Research Center for Supporting Software of Enterprise Internet Services, Shenzhen 518057, China
Kingdee Research, Kingdee International Software Group Company Limited, Shenzhen 518057, China
Tsinghua-CUHK Joint Research Center for Media Sciences, Technologies and Systems, Shenzhen Key Laboratory of Information Science and Technology, Graduate School at Shenzhen, Tsinghua University, Shenzhen 518055, China
Tsinghua National Laboratory for Information Science and Technology (TNList), Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
References (79)
Show source-provided references
1. Review of text-to-speech conversion for English10.1121/1.395275 ↗
2. Allen, J., Hunnicutt, M.S., Klatt, D.H., Armstrong, R.C., and Pisoni, D.B. (1987). From Text to Speech: The MITalk System, Cambridge University Press.
3. Emotional stress in synthetic speech: Progress and future directions10.1016/S0167-6393(96)00046-5 ↗
4. (2019, July 03). Festival. Available online: http://www.cstr.ed.ac.uk/projects/festival/.
5. Chu, M., Peng, H., and Zhao, Y. (2003, January 6–10). Microsoft Mulan. A bilingual TTS system. Proceedings of the 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’03), Hong Kong, China.
6. Speech synthesis based on hidden Markov models10.1109/JPROC.2013.2251852 ↗
7. Murray, I.R. (1989). Simulating Emotion in Synthetic Speech, University of Dundee.
8. Tokuda, K., Yoshimura, T., Masuko, T., Kobayashi, T., and Kitamura, T. (2000, January 5–9). Speech parameter generation algorithms for HMM-based speech synthesis. Proceedings of the 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing, Istanbul, Turkey.
9. Ratnaparkhi, A. (1997). A Simple Introduction to Maximum Entropy Models for Natural Language Processing, University of Pennsylvania.
10. Deep learning theory and its application in speech recognition
11. Graves, A. (2012). Supervised Sequence Labelling with Recurrent Neural Networks, Springer.10.1007/978-3-642-24797-2 ↗
12. Sutskever, I., Vinyals, O., and Le, Q.V. (2014, January 8–13). Sequence to sequence learning with neural networks. Proceedings of the Annual Conference on Neural Information Processing Systems, Montreal, QB, Canada.
13. Graves, A., Mohamed, A.R., and Hinton, G. (2013, January 26–31). Speech recognition with deep recurrent neural networks. Proceedings of the 38th IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6638947 ↗
14. Statistical parametric speech synthesis10.1016/j.specom.2009.04.004 ↗
15. Xu, S.H. (2007). Study on HMM-Based Chinese Speech Synthesis, Beijing University of Posts and Telecommunications.
16. Sotelo, J., Mehri, S., Kumar, K., Santos, J.F., Kastner, K., Courville, A., and Bengio, Y. (2017, January 24–26). Char2wav: End-to-end Speech Synthesis. Proceedings of the International Conference on Learning Representations Workshop, Toulon, France.
17. Software for a cascade/parallel formant synthesizer10.1121/1.383940 ↗
18. Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphone10.1016/0167-6393(90)90021-Z ↗
19. Yoshimura, T., Tokuda, K., Masuko, T., Kobayashi, T., and Kitamura, T. (1999, January 5–9). Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis. Proceedings of the Sixth European Conference on Speech Communication and Technology (EUROSPEECH’99), Budapest, Hungary.10.21437/Eurospeech.1999-513 ↗
20. Zen, H., Senior, A., and Schuster, M. (2013, January 26–31). Statistical parametric speech synthesis using deep neural networks. Proceedings of the 38th IEEE International Conference on Acoustics, Speech, and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6639215 ↗
21. Speech analysis and synthesis by linear prediction of the speech wave10.1121/1.1912679 ↗
22. Wu, Y.J. (2006). Study on the HMM-Based Speech Synthesis Techniques, University of Science and Technology of China.
23. Synthesis of voiced sounds using low-dimensional models of the vocal cords and time-varying subglottal pressure10.1016/j.mechrescom.2005.05.007 ↗
24. Schröder, M. (2001, January 2–7). Emotional speech synthesis: A review. Proceedings of the Seventh European Conference on Speech Communication and Technology, Aalborg, Denmark.10.21437/Eurospeech.2001-150 ↗
25. Zen, H., Nose, T., Yamagishi, J., Sako, S., Masuko, T., Black, A.W., and Tokuda, K. (2007, January 22–24). The HMM-based speech synthesis system (HTS) version 2.0. Proceedings of the ISCA Workshop on Speech Synthesis, Bonn, Germany.
26. Meng, F.B. (2013). Analysis and Generation of Focus in Continuous Speech, Tsinghua University.
27. Zhuang, X., Huang, J., Potamianos, G., and Hasegawa-Johnson, M. (2009, January 19–24). Acoustic fall detection using Gaussian mixture models and GMM supervectors. Proceedings of the 34th IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan.
28. Lin, Z.H. (2008). Research on Speech Synthesis Technology Based on Statistical Acoustic Modeling, University of Science and Technology of China.
29. Fan, Y., Qian, Y., Xie, F.L., and Soong, F.K. (2014, January 14–18). TTS synthesis with bidirectional LSTM based recurrent neural networks. Proceedings of the Fifteenth Annual Conference of the International Speech Communication Association, Singapore.10.21437/Interspeech.2014-443 ↗
30. Zen, H. (2015, January 19–20). Acoustic modeling in statistical parametric speech synthesis-from HMM to LSTM-RNN. Proceedings of the The First, International Workshop on Machine Learning in Spoken Language Processing (MLSLP2015), Aizu, Japan.
31. Remaking speech10.1121/1.1916020 ↗
32. Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based f0 extraction10.1016/S0167-6393(98)00085-5 ↗
33. (2015, March 15). HMM/DNN-Based Speech Synthesis System (HTS). Available online: http://hts.sp.nitech.ac.jp/.
34. Implementation of realtime STRAIGHT speech manipulation system: Report on its first implementation10.1250/ast.28.140 ↗
35. (2018, July 25). STRAIGHT. Available online: https://github.com/shuaijiang/STRAIGHT.
36. Phase vocoder10.1002/j.1538-7305.1966.tb01706.x ↗
37. Speech analysis/synthesis based on a sinusoidal representation10.1109/TASSP.1986.1164910 ↗
38. WORLD: A vocoder-based high-quality speech synthesis system for real-time applications10.1587/transinf.2015EDP7457 ↗
39. (2019, May 18). World. Available online: https://github.com/mmorise/World.
40. Ling, Z.H., Deng, L., and Yu, D. (2013, January 26–31). Modeling spectral envelopes using restricted Boltzmann machines for statistical parametric speech synthesis. Proceedings of the 38th IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6639187 ↗
41. Deng, L., Seltzer, M.L., Yu, D., Acero, A., Mohamed, A.R., and Hinton, G. (2010, January 26–30). Binary coding of speech spectrograms using a deep auto-encoder. Proceedings of the Eleventh Annual Conference of the International Speech Communication Association, Makuhari, Japan.10.21437/Interspeech.2010-487 ↗
42. Gehring, J., Miao, Y., Metze, F., and Waibel, A. (2013, January 26–31). Extracting deep bottleneck features using stacked auto-encoders. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6638284 ↗
43. Kang, S.Y., Qian, X.J., and Meng, H. (2013, January 26–31). Multi-distribution deep belief network for speech synthesis. Proceedings of the 38th IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6639225 ↗
44. Zen, H., and Senior, A. (2014, January 4–9). Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis. Proceedings of the 39th IEEE International Conference on Acoustics, Speech and Signal Processing, Florence, Italy.10.1109/ICASSP.2014.6854321 ↗
45. Bishop, C. (1994). Mixture Density Networks, Neural Computing Research Group, Aston University. Tech. Rep. NCRG/94/004.
46. Graves, A., and Schmidhuber, J. (August, January 31). Framewise phoneme classification with bidirectional LSTM networks. Proceedings of the IEEE International Joint Conference on Neural Networks, Montreal, QC, Canada.
47. Graves, A., Fernandez, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist temporal classification: Labeling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143891 ↗
48. Li, R.N., Wu, Z.Y., Liu, X.Y., Meng, H., and Cai, L.H. (2017, January 5–9). Multi-task learning of structured output layer bidirectional LSTMs for speech synthesis. Proceedings of the 42nd IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, LA, USA.10.1109/ICASSP.2017.7953210 ↗
49. Yu, Q.J., Liu, P., Wu, Z.Y., Kang, S.Y., Meng, H., and Cai, L.H. (2016, January 20–25). Learning cross-lingual information with multilingual BLSTM for speech synthesis of low-resource languages. Proceedings of the 41st IEEE International Conference on Acoustics, Speech and Signal Processing, Shanghai, China.10.1109/ICASSP.2016.7472738 ↗
50. Caruana, R. (1998). Multitask Learning, Springer.10.1007/978-1-4615-5529-2_5 ↗
51. Seltzer, M.L., and Droppo, J. (2013, January 26–31). Multi-task learning in deep neural networks for improved phoneme recognition. Proceedings of the 38th IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6639012 ↗
52. Luong, M.T., Pham, H., and Manning, C.D. (2015). Effective approaches to attention-based neural machine translation. arXiv.10.18653/v1/D15-1166 ↗
53. Jaitly, N., Le, Q.V., Vinyals, O., Sutskever, I., Sussillo, D., and Bengio, S. (2016, January 5–10). An online sequence-to-sequence model using partial conditioning. Proceedings of the Annual Conference on Neural Information Processing Systems, Barcelona, Spain.
54. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the 32nd International Conference on Machine Learning, Lille, France.
55. Wang, W., Xu, S., and Xu, B. (2016, January 8–12). First step towards end-to-end parametric TTS synthesis: Generating spectral parameters with neural attention. Proceedings of the Seventeenth Annual Conference of the International Speech Communication Association, San Francisco, CA, USA.10.21437/Interspeech.2016-134 ↗
56. Bahdanau, D., and Cho, K. (2014). ; Bengio. Neural machine translation by jointly learning to align and translate. arXiv.
57. Zhang, J.X., Ling, Z.H., and Dai, L.R. (2018, January 15–20). Forward attention in sequence-to-sequence acoustic modeling for speech synthesis. Proceedings of the 43rd IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada.10.1109/ICASSP.2018.8462020 ↗
58. Oord, A.V.D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., and Kavukcuoglu, K. (2017). WaveNet: A generative model for raw audio. arXiv Preprint.
59. Oord, A.V.D., Kalchbrenner, N., Espeholt, L., Vinyals, O., and and Graves, A. (2016, January 5–10). Conditional image generation with pixelcnn decoders. Proceedings of the Annual Conference on Neural Information Processing Systems, Barcelona, Spain.
60. Oord, A.V.D., Kalchbrenner, N., and Kavukcuoglu, K. (2016). Pixel recurrent neural networks. arXiv.
61. Oord, A.V.D., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., and Casagrande, N. (2017). Parallel wavenet: Fast high-fidelity speech synthesis. arXiv.
62. Arik, S.Ö., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., and Sengupta, S. (2017, January 6–11). Deep voice: Real-time neural text-to-speech. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia.
63. Wang, Y., Skerry-Ryan, R.J., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., and Le, Q. (2017). Tacotron: Towards end-to-end speech synthesis. arXiv.10.21437/Interspeech.2017-1452 ↗
64. Wang, Y., Skerry-Ryan, R.J., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., and Le, Q.V. (2017). Tacotron: A fully end-to-end text-to-speech synthesis model. arXiv.
65. Signal estimation from modified short-time Fourier transform10.1109/TASSP.1984.1164317 ↗
66. Barron, A. (2018, October 20). Implementation of Google’s Tacotron in TensorFlow. Available online: https://github.com/Kyubyong/tacotron.
67. Ito, K. (2018, October 20). Tacotron Speech Synthesis Implemented in TensorFlow, with Samples and a Pre-Trained Model. Available online: https://github.com/keithito/tacotron.
68. Yamamoto, R. (2018, October 20). PyTorch Implementation of Tacotron Speech Synthesis Model. Available online: https://github.com/r9y9/tacotron_pytorch.
69. Tachibana, H., Uenoyama, K., and Aihara, S. (2018, January 15–20). Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention. Proceedings of the 43rd IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada.10.1109/ICASSP.2018.8461829 ↗
70. Zhang, Y.J., Pan, S., He, L., and Ling, Z.H. (2019, January 12–17). Learning latent representations for style control and transfer in end-to-end speech synthesis. Proceedings of the 44th IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK.10.1109/ICASSP.2019.8683623 ↗
71. Kingma, D.P., and Welling, M. (2014). Auto-encoding variational bayes. arXiv.
72. Gibiansky, A., Arik, S., Diamos, G., Miller, J., Peng, K., Ping, W., and Zhou, Y. (2017, January 4–9). Deep voice 2: Multi-speaker neural text-to-speech. Proceedings of the Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA.
73. Shen, J., Pang, R., Weiss, R.J., Schuster, M., Jaitly, N., Yang, Z., and Saurous, R.A. (2018, January 15–20). Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. Proceedings of the 43rd IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada.10.1109/ICASSP.2018.8461368 ↗
74. Yasuda, Y., Wang, X., Takaki, S., and Yamagishi, J. (2019, January 12–17). Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent language. Proceedings of the 44th IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK.10.1109/ICASSP.2019.8682353 ↗
75. Nakamura, K., Hashimoto, K., Oura, K., Nankaku, Y., and Tokuda, K. (2019). Singing voice synthesis based on convolutional neural networks. arXiv.10.1109/ICASSP40776.2020.9053811 ↗
76. Ping, W., Peng, K., Gibiansky, A., Arik, S.O., Kannan, A., and Narang, S. (May, January April). Deep voice 3: Scaling text-to-speech with convolutional sequence learning. Proceedings of the Sixth International Conference on Learning Representations, Vancouver, BC, Canada.
77. A survey on dialogue systems: Recent advances and new frontiers10.1145/3166054.3166058 ↗
78. Mametani, K., Kato, T., and Yamamoto, S. (2019, January 12–17). Investigating context features hidden in End-to-End TTS. Proceedings of the IEEE the 44th International Conference on Acoustics, Speech and Signal Processing, Brighton, UK.10.1109/ICASSP.2019.8683857 ↗
79. Chung, Y.A., Wang, Y., Hsu, W.N., Zhang, Y., and Skerry-Ryan, R.J. (2019, January 12–17). Semi-supervised training for improving data efficiency in end-to-end speech synthesis. Proceedings of the 44th IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK.10.1109/ICASSP.2019.8683862 ↗