Overview of Voice Conversion Methods Based on Deep Learning
Source abstract
Abstract
Voice conversion is a process where the essence of a speaker’s identity is seamlessly transferred to another speaker, all while preserving the content of their speech. This usage is accomplished using algorithms that blend speech processing techniques, such as speech analysis, speaker classification, and vocoding. The cutting-edge voice conversion technology is characterized by deep neural networks that effectively separate a speaker’s voice from their linguistic content. This article offers a comprehensive overview of the development status of this area of science based on the current state-of-the-art voice conversion methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Communication Systems, Faculty of Electronics, Military University of Technology, 00-908 Warsaw, Poland
Funding
National Centre for Research and Development
References (66)
Show source-provided references
1. Voice conversion10.1016/0167-6393(89)90041-1 ↗
2. An Overview of Voice Conversion Systems10.1016/j.specom.2017.01.008 ↗
3. An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning10.1109/TASLP.2020.3038524 ↗
4. Variani, E., Lei, X., McDermott, E., Moreno, I.L., and Gonzalez-Dominguez, J. (2014, January 4–9). Deep Neural Networks for Small Footprint Text-dependent Speaker Verification. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854363 ↗
5. Voice conversion versus speaker verification: An overview10.1017/ATSIP.2014.17 ↗
6. van den Unsupervised speech representation learning using WaveNet autoencoders10.1109/TASLP.2019.2938863 ↗
7. Kashkin, A., Karpukhin, I., and Shishkin, S. (2022). HiFi-VC: High Quality ASR-Based Voice Conversion. arXiv.
8. Chen, M., Zhou, Y., Huang, H., and Hain, T. (2022). Efficient Non-Autoregressive GAN Voice Conversion using VQWav2vec Features and Dynamic Convolution. arXiv.
9. Baevski, A., Schneider, S., and Auli, M. (2019). vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations. arXiv.
10. Wu, F., Fan, A., Baevski, A., Dauphin, Y.N., and Auli, M. (2019). Pay Less Attention with Lightweight and Dynamic Convolutions. arXiv.
11. Siuzdak, H., Dura, P., van Rijn, P., and Jacoby, N. (2022). WavThruVec: Latent speech representation as intermediate features for neural speech synthesis. arXiv.10.21437/Interspeech.2022-10797 ↗
12. Kingma, D.P., and Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv.
13. Long, Z., Zheng, Y., Yu, M., and Xin, J. (2022). Enhancing Zero-Shot Many to Many Voice Conversion with Self-Attention VAE. arXiv.
14. Lian, J., Zhang, C., and Yu, D. (2022). Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversion. arXiv.10.1109/ICASSP43922.2022.9747272 ↗
15. Li, Y.A., Zare, A., and Mesgarani, N. (2021). StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion. arXiv.10.21437/Interspeech.2021-319 ↗
16. Li, J., Tu, W., and Xiao, L. (2022). FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion. arXiv.
17. Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
18. Wan, L., Wang, Q., Papir, A., and Moreno, I.L. (2017). Generalized End-to-End Loss for Speaker Verification. arXiv.10.1109/ICASSP.2018.8462665 ↗
19. Jia, Y., Zhang, Y., Weiss, R.J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., and Moreno, I.L. (2018). Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis. arXiv.
20. Qian, K., Zhang, Y., Chang, S., Yang, X., and Hasegawa-Johnson, M. (2019). AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss. arXiv.
21. Vaswani, A., Brain, G., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. arXiv.
22. Desplanques, B., Thienpondt, J., and Demuynck, K. (2020). ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. arXiv.10.21437/Interspeech.2020-2650 ↗
23. Yang, B., Lyu, J., Zhang, S., Qi, Y., and Xin, J. (2019, January 25–27). Channel pruning for deep neural networks via a relaxed groupwise splitting method. Proceedings of the 2019 2nd International Conference on Artificial Intelligence for Industries, AI4I 2019, Lagana Hills, CA, USA.10.1109/AI4I46381.2019.00032 ↗
24. Chou, J., Yeh, C., and Lee, H. (2019). One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization. arXiv.10.21437/Interspeech.2019-2663 ↗
25. Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling10.1109/TASLP.2021.3076867 ↗
26. Polyak, A., Wolf, L., and Taigman, Y. (2019). TTS Skins: Speaker Conversion via ASR. arXiv.10.21437/Interspeech.2020-1416 ↗
27. Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., and Wu, Y. (2019). Conformer: Convolution-augmented Transformer for Speech Recognition. arXiv.10.21437/Interspeech.2020-3015 ↗
28. Schneider, S., Baevski, A., Collobert, R., and Auli, M. (2019). wav2vec: Unsupervised Pre-training for Speech Recognition. arXiv.10.21437/Interspeech.2019-1873 ↗
29. Kum, S., and Nam, J. (2019). Joint detection and classification of singing voice melody using convolutional recurrent neural networks. Appl. Sci., 9.10.3390/app9071324 ↗
30. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing10.1109/JSTSP.2022.3188113 ↗
31. Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2016). Density estimation using Real NVP. arXiv.
32. Huang, X., and Belongie, S. (2017). Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. arXiv.10.1109/ICCV.2017.167 ↗
33. Yamamoto, R., Song, E., and Kim, J.-M. (2019). Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. arXiv.10.1109/ICASSP40776.2020.9053795 ↗
34. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2019). Analyzing and Improving the Image Quality of StyleGAN. arXiv.10.1109/CVPR42600.2020.00813 ↗
35. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
36. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). WaveNet: A Generative Model for Raw Audio. arXiv.
37. Nguyen, B., and Cardinaux, F. (2021). NVC-Net: End-to-End Adversarial Voice Conversion. arXiv.10.1109/ICASSP43922.2022.9747020 ↗
38. Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A., and Deepmind, G. (2017, January 24–26). β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. Proceedings of the 2017 International Conference on Learning Representations, Toulon, France.
39. Shen, J., Pang, R., Weiss, R.J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., and Skerry-Ryan, R. (2018, January 15–20). Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461368 ↗
40. Li, Y., and Mandt, S. (2018). Disentangled Sequential Autoencoder. arXiv.
41. van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016). Pixel Recurrent Neural Networks. arXiv.
42. Holschneider, M., Kronland-Martinet, R., Morlet, J., and Tchamitchian, P. (1990). Wavelets: Time-Frequency Methods and Phase Space, Proceedings of the International Conference, Marseille, France, 14–18 December 1987, Springer.
43. Dutilleux, P. (1989). Wavelets: Time-Frequency Methods and Phase Space, Proceedings of the International Conference, Marseille, France, 14–18 December 1987, Springer.
44. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2014). Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. arXiv.
45. Yu, F., and Koltun, V. (2015). Multi-Scale Context Aggregation by Dilated Convolutions. arXiv.
46. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel.
47. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.10.1109/CVPR.2016.90 ↗
48. Generative Adversarial Networks: An overview10.1109/MSP.2017.2765202 ↗
49. Yamamoto, R., Song, E., and Kim, J.M. (2019, January 15–19). Probability density distillation with generative adversarial networks for high-quality parallel waveform generation. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, Graz, Austria.10.21437/Interspeech.2019-1965 ↗
50. Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks10.1109/LSP.2018.2880284 ↗
51. Wang, X., Takaki, S., and Yamagishi, J. (2018). Neural source-filter-based waveform model for statistical parametric speech synthesis. arXiv.10.1109/ICASSP.2019.8682298 ↗
52. Wang, C., Zeng, C., and He, X. (2022). HiFi-WaveGAN: Generative Adversarial Network with Auxiliary Spectrogram-Phase Loss for High-Fidelity Singing Voice Generation. arXiv.
53. Donahue, C., McAuley, J., and Puckette, M. (2018). Adversarial Audio Synthesis. arXiv.
54. Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W.Z., Sotelo, J., de Brebisson, A., Bengio, Y., and Courville, A. (2019). MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis. arXiv.
55. Liu, Z., and Mak, B. (2019). Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers. arXiv.10.21437/Interspeech.2020-1464 ↗
56. Bakhturina, E., Lavrukhin, V., Ginsburg, B., and Zhang, Y. (2021). Hi-Fi Multi-Speaker English TTS Dataset. arXiv.10.21437/Interspeech.2021-1599 ↗
57. Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. (2015, January 19–24). Librispeech: An ASR corpus based on public domain audio books. Proceedings of the ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing, Brisbane, QLD, Australia.10.1109/ICASSP.2015.7178964 ↗
58. Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F.M., and Weber, G. (2019). Common Voice: A Massively-Multilingual Speech Corpus. arXiv.
59. Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W.T., and Rubinstein, M. (2018). Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation. arXiv.10.1145/3197517.3201357 ↗
60. Garofolo, J.S. (1993). TIMIT Acoustic-Phonetic Continuous Speech Corpus LDC93S1. Linguist. Data Consort.
61. Zhao, Y., Huang, W.-C., Tian, X., Yamagishi, J., Das, R.K., Kinnunen, T., Ling, Z., and Toda, T. (2020). Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion. arXiv.
62. Takamichi, S., Mitsui, K., Saito, Y., Koriyama, T., Tanji, N., and Saruwatari, H. (2019). JVS corpus: Free Japanese multi-speaker voice corpus. arXiv.
63. Zhou, K., Sisman, B., Liu, R., and Li, H. (2020). Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset. arXiv.10.1109/ICASSP39728.2021.9413391 ↗
64. Lo, C.-C., Fu, S.-W., Huang, W.-C., Wang, X., Yamagishi, J., Tsao, Y., and Wang, H.-M. (2019). MOSNet: Deep Learning based Objective Assessment for Voice Conversion. arXiv.10.21437/Interspeech.2019-2003 ↗
65. Lian, J., Zhang, C., Anumanchipalli, G.K., and Yu, D. (2022). Towards Improved Zero-shot Voice Conversion with Conditional DSVAE. arXiv.10.21437/Interspeech.2022-11225 ↗
66. Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R.J., Jia, Y., Chen, Z., and Wu, Y. (2019). LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. arXiv.10.21437/Interspeech.2019-2441 ↗