A Neural Parametric Singing Synthesizer Modeling Timbre and Expression from Natural Songs
Source abstract
Abstract
We recently presented a new model for singing synthesis based on a modified version of the WaveNet architecture. Instead of modeling raw waveform, we model features produced by a parametric vocoder that separates the influence of pitch and timbre. This allows conveniently modifying pitch to match any target melody, facilitates training on more modest dataset sizes, and significantly reduces training and generation times. Nonetheless, compared to modeling waveform directly, ways of effectively handling higher-dimensional outputs, multiple feature streams and regularization become more important with our approach. In this work, we extend our proposed system to include additional components for predicting F0 and phonetic timings from a musical score with lyrics. These expression-related features are learned together with timbrical features from a single set of natural songs. We compare our method to existing statistical parametric, concatenative, and neural network-based approaches using quantitative metrics as well as listening tests.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Music Technology and Sound Studies · Speech and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Music Technology Group, Universitat Pompeu Fabra, 08012 Barcelona, Spain
References (43)
Show source-provided references
1. Bonada, J., Umbert, M., and Blaauw, M. (2016, January 8–12). Expressive singing synthesis based on unit selection for the singing synthesis challenge 2016. Proceedings of the 17th Annual Conference of the International Speech Communication Association (Interspeech), San Francisco, CA, USA.10.21437/Interspeech.2016-872 ↗
2. Synthesis of the Singing Voice by Performance Sampling and Spectral Models10.1109/MSP.2007.323266 ↗
3. Saino, K., Zen, H., Nankaku, Y., Lee, A., and Tokuda, K. (2006, January 17–21). An HMM-based singing voice synthesis system. Proceedings of the 9th International Conference on Spoken Language Processing (ICSLP—Interspeech), Pittsburgh, PA, USA.10.21437/Interspeech.2006-584 ↗
4. Oura, K., Mase, A., Yamada, T., Muto, S., Nankaku, Y., and Tokuda, K. (2010, January 22–24). Recent development of the HMM-based singing voice synthesis system—Sinsy. Proceedings of the 7th ISCA Workshop on Speech Synthesis (SSW7), Kyoto, Japan.
5. Van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A.W., and Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. CoRR arXiv.
6. Blaauw, M., and Bonada, J. (2017, January 20–24). A neural parametric singing synthesizer. Proceedings of the 18th Annual Conference of the International Speech Communication Association (Interspeech), Stockholm, Sweden.10.21437/Interspeech.2017-1420 ↗
7. Yu, F., and Koltun, V. (2016, January 2–4). Multi-Scale Context Aggregation by Dilated Convolutions. Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
8. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the 34th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
9. Reed, S., van den Oord, A., Kalchbrenner, N., Bapst, V., Botvinick, M., and de Freitas, N. (2016). Generating Interpretable Images with Controllable Structure, Google DeepMind. Technical Report.
10. Van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., and Kavukcuoglu, K. (2016, January 5–10). Conditional image generation with PixelCNN decoders. Proceedings of the Advances in Neural Information Processing Systems 29 (NIPS), Barcelona, Spain.
11. Van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016, January 19–24). Pixel recurrent neural networks. Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, NY, USA.
12. Salimans, T., Karpathy, A., Chen, X., and Kingma, D.P. (2017, January 24–26). PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications. Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France.
13. Blaauw, M., and Bonada, J. (2017, October 01). A Singing Synthesizer Based on PixelCNN. Available online: http://www.dtic.upf.edu/~mblaauw/MdM_NIPS_seminar/.
14. Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. (2016, January 2–4). Sequence level training with recurrent neural networks. Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
15. Wang, X., Takaki, S., and Yamagishi, J. (2017, January 20–24). A RNN-based quantized F0 model with multi-tier feedback links for text-to-speech synthesis. Proceedings of the 18th Annual Conference of the International Speech Communication Association (Interspeech), Stockholm, Sweden.10.21437/Interspeech.2017-246 ↗
16. Taylor, P. (2009). Text-to-Speech Synthesis, Cambridge University Press. Chapter 9.1.4.10.1017/CBO9780511816338 ↗
17. Umbert, M., Bonada, J., and Blaauw, M. (August, January 30). Generating singing voice expression contours based on unit selection. Proceedings of the 4th Stockholm Music Acoustics Conference (SMAC), Stockholm, Sweden.
18. Mase, A., Oura, K., Nankaku, Y., and Tokuda, K. (2010, January 26–30). HMM-based singing voice synthesis system using pitch-shifted pseudo training data. Proceedings of the 11th Annual Conference of the International Speech Communication Association (Interspeech), Makuhari, Chiba, Japan.10.21437/Interspeech.2010-188 ↗
19. Nakamura, K., Oura, K., Nankaku, Y., and Tokuda, K. (2014, January 4–9). HMM-based singing voice synthesis and its application to Japanese and English. Proceedings of the 39th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6853599 ↗
20. Arik, S.Ö., Diamos, G., Gibiansky, A., Miller, J., Peng, K., Ping, W., Raiman, J., and Zhou, Y. (2017, January 4–9). Deep voice 2: Multi-speaker neural text-to-speech. Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS), Long Beach, CA, USA.
21. WORLD: A vocoder-based high-quality speech synthesis system for real-time applications10.1587/transinf.2015EDP7457 ↗
22. D4C, a band-aperiodicity estimator for high-quality speech synthesis10.1016/j.specom.2016.09.001 ↗
23. Tokuda, K., Kobayashi, T., Masuko, T., and Imai, S. (1994, January 18–22). Mel-generalized cepstral analysis—A unified approach to speech spectral estimation. Proceedings of the 3rd International Conference on Spoken Language Processing (ICSLP), Yokohama, Japan.10.21437/ICSLP.1994-275 ↗
24. Deterministic annealing EM algorithm10.1016/S0893-6080(97)00133-0 ↗
25. Ramachandran, P., Paine, T.L., Khorrami, P., Babaeizadeh, M., Chang, S., Zhang, Y., Hasegawa-Johnson, M., Campbell, R., and Huang, T. (2017, January 24–26). Fast generation for convolutional autoregressive models. Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France.
26. Arik, S.Ö., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Raiman, J., and Sengupta, S. (2017, January 10–15). Deep voice: Real-time neural text-to-speech. Proceedings of the 34th International Conference on Machine Learning (ICML), Stockholm, Sweden.
27. Kalchbrenner, N., van den Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. (2016). Video pixel networks. CoRR arXiv.
28. Kalchbrenner, N., Espeholt, L., Simonyan, K., van den Oord, A., Graves, A., and Kavukcuoglu, K. (2016). Neural machine translation in linear time. CoRR arXiv.
29. Mehri, S., Kumar, K., Gulrajani, I., Kumar, R., Jain, S., Sotelo, J., Courville, A.C., and Bengio, Y. (2017, January 24–26). SampleRNN: An unconditional end-to-end neural audio generation model. Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France.
30. Sotelo, J., Mehri, S., Kumar, K., Santos, J.F., Kastner, K., Courville, A., and Bengio, Y. (2017, January 24–26). Char2Wav: End-to-End speech synthesis. Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France.
31. Wang, Y., Skerry-Ryan, R.J., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., and Bengio, S. (2017, January 20–24). Tacotron: A fully end-to-end text-to-speech synthesis model. Proceedings of the 18th Annual Conference of the International Speech Communication Association (Interspeech), Stockholm, Sweden.
32. Zen, H., and Senior, A. (2014, January 4–9). Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis. Proceedings of the 39th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854321 ↗
33. Zen, H., and Sak, H. (2015, January 19–24). Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis. Proceedings of the 40th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), South Brisbane, QLD, Australia.10.1109/ICASSP.2015.7178816 ↗
34. Tokuda, K., Yoshimura, T., Masuko, T., Kobayashi, T., and Kitamura, T. (2000, January 5–9). Speech parameter generation algorithms for HMM-based speech synthesis. Proceedings of the 25th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Istanbul, Turkey.
35. Oura, K., Mase, A., Nankaku, Y., and Tokuda, K. (2012, January 25–30). Pitch adaptive training for HMM-based singing voice synthesis. Proceedings of the 37th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan.10.1109/ICASSP.2012.6289136 ↗
36. Shirota, K., Nakamura, K., Hashimoto, K., Oura, K., Nankaku, Y., and Tokuda, K. (2014, January 4–9). Integration of speaker and pitch adaptive training for HMM-based singing voice synthesis. Proceedings of the 39th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854062 ↗
37. Nishimura, M., Hashimoto, K., Oura, K., Nankaku, Y., and Tokuda, K. (2016, January 8–12). Singing voice synthesis based on deep neural networks. Proceedings of the 17th Annual Conference of the International Speech Communication Association (Interspeech), San Francisco, CA, USA.10.21437/Interspeech.2016-1027 ↗
38. Zen, H., Nose, T., Yamagishi, J., Sako, S., Masuko, T., Black, A.W., and Tokuda, K. (2007, January 22–24). The HMM-based speech synthesis system (HTS) version 2.0. Proceedings of the 6th ISCA Workshop on Speech Synthesis (SSW6), Bonn, Germany.
39. Iglewicz, B., and Hoaglin, D.C. (1993). How to Detect and Handle Outliers, ASQC Quality Press. ASQC Basic References in Quality Control.
40. Postfilters to modify the modulation spectrum for statistical parametric speech synthesis10.1109/TASLP.2016.2522655 ↗
41. Expression control in singing voice synthesis: Features, approaches, evaluation, and challenges10.1109/MSP.2015.2424572 ↗
42. ITU-R Recommendation BS.1534-3 (2015). Method for the Subjective Assessment of Intermediate Quality Levels of Coding Systems, International Telecommunication Union. Technical Report.
43. Kingma, D.P., and Ba, J.L. (2015, January 7–9). Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA.