MPFM-VC: A Voice Conversion Algorithm Based on Multi-Dimensional Perception Flow Matching
Source abstract
Abstract
Voice conversion (VC) is an advanced technology that enables the transformation of raw speech into high-quality audio resembling the target speaker’s voice while preserving the original linguistic content and prosodic patterns. In this study, we propose a voice conversion algorithm, Multi-Dimensional Perception Flow Matching (MPFM-VC). Unlike traditional approaches that directly generate waveform outputs, MPFM-VC models the evolutionary trajectory of mel spectrograms with a flow-matching framework and incorporates a multi-dimensional feature perception network to enhance the stability and quality of speech synthesis. Additionally, we introduce a content perturbation method during training to improve the model’s generalization ability and reduce inference-time artifacts. To further increase speaker similarity, an adversarial training mechanism on speaker embeddings is employed to achieve effective disentanglement between content and speaker identity representations, thereby enhancing the timbre consistency of the converted speech. Experimental results for both speech and singing voice conversion tasks show that MPFM-VC achieves competitive performance compared to existing state-of-the-art VC models in both subjective and objective evaluation metrics. The synthesized speech shows improved naturalness, clarity, and timbre fidelity in both objective and subjective evaluations, suggesting the potential effectiveness of the proposed approach.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Technology, Jinan University, Guangzhou 510632, China
School of Mathematics and Statistics, Changchun University of Technology, Changchun 130012, China
Funding
Personal
References (53)
Show source-provided references
1. Conventional and contemporary approaches used in text to speech synthesis: A review10.1007/s10462-022-10315-0 ↗
2. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.Y. (2019). Fastspeech: Fast, robust and controllable text to speech. Advances in Neural Information Processing Systems, Proceedings of the Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Vancouver, BC, Canada, 8–14 December 2019, Curran Associates Inc.
3. Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.Y. (2020). Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv.
4. Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., and Bengio, S. (2017). Tacotron: Towards end-to-end speech synthesis. arXiv.10.21437/Interspeech.2017-1452 ↗
5. Shen, J., Pang, R., Weiss, R.J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., and Skerrv-Ryan, R. (2018, January 15–20). Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461368 ↗
6. Li, N., Liu, S., Liu, Y., Zhao, S., and Liu, M. (February, January 27). Neural speech synthesis with transformer network. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
7. Saito, Y., Ijima, Y., Nishida, K., and Takamichi, S. (2018, January 15–20). Non-parallel voice conversion using variational autoencoders conditioned by phonetic posteriorgrams and d-vectors. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461384 ↗
8. Qian, K., Zhang, Y., Chang, S., Yang, X., and Hasegawa-Johnson, M. (2019, January 9–15). Autovc: Zero-shot voice style transfer with only autoencoder loss. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
9. Chou, J.c., Yeh, C.c., and Lee, H.y. (2019). One-shot voice conversion by separating speaker and content representations with instance normalization. arXiv.10.21437/Interspeech.2019-2663 ↗
10. Jia, Y., Zhang, Y., Weiss, R., Wang, Q., Shen, J., Ren, F., Nguyen, P., Pang, R., Lopez Moreno, I., and Wu, Y. (2018). Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances in Neural Information Processing Systems, Proceedings of the Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, QC, Canada, 3–8 December 2018, Curran Associates Inc.
11. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023, January 23–29). Robust speech recognition via large-scale weak supervision. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA.
12. Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, Proceedings of the Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Online, 6–12 December 2020, Curran Associates Inc.
13. Qian, K., Zhang, Y., Gao, H., Ni, J., Lai, C.I., Cox, D., Hasegawa-Johnson, M., and Chang, S. (2022, January 17–23). Contentvec: An improved self-supervised speech representation by disentangling speakers. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA.
14. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
15. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, Proceedings of the Annual Conference on Neural Information Processing Systems 2014, NeurIPS 2014, Montréal, QC, Canada, 8–13 December 2014, Curran Associates Inc.
16. Rezende, D.J., Mohamed, S., and Wierstra, D. (2014, January 21–26). Stochastic backpropagation and approximate inference in deep generative models. Proceedings of the International Conference on Machine Learning, Beijing, China.
17. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, Proceedings of the Annual Conference on Neural Information Processing Systems 2017, NeurIPS 2017, Long Beach, CA, USA, 4–9 December 2017, Curran Associates Inc.
18. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015, January 6–11). Deep unsupervised learning using nonequilibrium thermodynamics. Proceedings of the International Conference on Machine Learning, Lille, France.
19. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis
20. Lee, S.g., Ping, W., Ginsburg, B., Catanzaro, B., and Yoon, S. (2022). Bigvgan: A universal neural vocoder with large-scale training. arXiv.
21. Kaneko, T., Tanaka, K., Kameoka, H., and Seki, S. (2022, January 22–27). iSTFTNet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time Fourier transform. Proceedings of the ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746713 ↗
22. Kim, J., Kong, J., and Son, J. (2021, January 18–24). Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. Proceedings of the International Conference on Machine Learning, Online.
23. Kong, J., Park, J., Kim, B., Kim, J., Kong, D., and Kim, S. (2023). Vits2: Improving quality and efficiency of single-stage text-to-speech with adversarial learning and architecture design. arXiv.10.21437/Interspeech.2023-534 ↗
24. Li, J., Tu, W., and Xiao, L. (2023, January 4–10). Freevc: Towards high-quality text-free one-shot voice conversion. Proceedings of the ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095191 ↗
25. Lei, Y., Yang, S., Cong, J., Xie, L., and Su, D. (2022). Glow-wavegan 2: High-quality zero-shot text-to-speech synthesis and any-to-any voice conversion. arXiv.10.21437/Interspeech.2022-684 ↗
26. Pankov, V., Pronina, V., Kuzmin, A., Borisov, M., Usoltsev, N., Zeng, X., Golubkov, A., Ermolenko, N., Shirshova, A., and Matveeva, Y. (2023). DINO-VITS: Data-Efficient Zero-Shot TTS with Self-Supervised Speaker Verification Loss for Noise Robustness. arXiv.10.21437/Interspeech.2024-549 ↗
27. Huang, W.C., Violeta, L.P., Liu, S., Shi, J., and Toda, T. (2023, January 16–20). The singing voice conversion challenge 2023. Proceedings of the 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Taipei, Taiwan.10.1109/ASRU57964.2023.10389671 ↗
28. Zhou, Y., Chen, M., Lei, Y., Zhu, J., and Zhao, W. (2023, January 16–20). VITS-based Singing Voice Conversion System with DSPGAN post-processing for SVCC2023. Proceedings of the 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Taipei, Taiwan.10.1109/ASRU57964.2023.10389649 ↗
29. Ning, Z., Jiang, Y., Wang, Z., Zhang, B., and Xie, L. (2023, January 16–20). Vits-based singing voice conversion leveraging whisper and multi-scale f0 modeling. Proceedings of the 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Taipei, Taiwan.10.1109/ASRU57964.2023.10389740 ↗
30. Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. (2020). Diffwave: A versatile diffusion model for audio synthesis. arXiv.
31. Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M. (2021, January 18–24). Grad-tts: A diffusion probabilistic model for text-to-speech. Proceedings of the International Conference on Machine Learning, Online.
32. Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., Kudinov, M.S., and Wei, J. (2022, January 25–29). Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme. Proceedings of the International Conference on Learning Representations, Online.
33. Choi, H.Y., Lee, S.H., and Lee, S.W. (2024, January 20–27). Dddm-vc: Decoupled denoising diffusion models with disentangled representation and prior mixup for verified robust voice conversion. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i16.29740 ↗
34. Ye, Z., Xue, W., Tan, X., Chen, J., Liu, Q., and Guo, Y. (November, January 29). Comospeech: One-step speech and singing voice synthesis via consistency model. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
35. Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., and Boesel, F. (2024, January 21–27). Scaling rectified flow transformers for high-resolution image synthesis. Proceedings of the Forty-First International Conference on Machine Learning, Vienna, Austria.
36. Le, M., Vyas, A., Shi, B., Karrer, B., Sari, L., Moritz, R., Williamson, M., Manohar, V., Adi, Y., and Mahadeokar, J. (2023). Voicebox: Text-guided multilingual universal speech generation at scale. Advances in Neural Information Processing Systems, Proceedings of the Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, 10–16 December 2023, Curran Associates Inc.
37. Guo, Y., Du, C., Ma, Z., Chen, X., and Yu, K. (2024, January 14–19). Voiceflow: Efficient text-to-speech with rectified flow matching. Proceedings of the ICASSP 2024–2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10445948 ↗
38. Guo, W., Zhang, Y., Pan, C., Huang, R., Tang, L., Li, R., Hong, Z., Wang, Y., and Zhao, Z. (2025). TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching. arXiv.10.1609/aaai.v39i22.34571 ↗
39. Sadekova, T., Kudinov, M., Popov, V., Yermekova, A., and Khrapov, A. (2024, January 1–5). PitchFlow: Adding pitch control to a Flow-matching based TTS model. Proceedings of the Interspeech 2024, Kos Island, Greece.10.21437/Interspeech.2024-1023 ↗
40. Yao, J., Yuguang, Y., Pan, Y., Ning, Z., Ye, J., Zhou, H., and Xie, L. (March, January 27). Stablevc: Style controllable zero-shot voice conversion with conditional flow matching. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA.
41. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. Lecture Notes in Computer Science, Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015, Proceedings Part III 18; Springer.
42. Wei, H., Cao, X., Dan, T., and Chen, Y. (2023). RMVPE: A robust model for vocal pitch estimation in polyphonic music. arXiv.10.21437/Interspeech.2023-528 ↗
43. Van Niekerk, B., Carbonneau, M.A., Zaïdi, J., Baas, M., Seuté, H., and Kamper, H. (2022, January 22–27). A comparison of discrete and soft speech units for improved voice conversion. Proceedings of the ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746484 ↗
44. Wang, H., Zheng, S., Chen, Y., Cheng, L., and Chen, Q. (2023). Cam++: A fast and efficient network for speaker verification using context-aware masking. arXiv.10.21437/Interspeech.2023-1513 ↗
45. Ho, J., and Salimans, T. (2022). Classifier-free diffusion guidance. arXiv.
46. An, K., Chen, Q., Deng, C., Du, Z., Gao, C., Gao, Z., Gu, Y., He, T., Hu, H., and Hu, K. (2024). Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and llms. arXiv.
47. Kameoka, H., Kaneko, T., Tanaka, K., and Hojo, N. (2018, January 18–21). Stargan-vc: Non-parallel many-to-many voice conversion using star generative adversarial networks. Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.10.1109/SLT.2018.8639535 ↗
48. Kaneko, T., and Kameoka, H. (2018, January 3–7). Cyclegan-vc: Non-parallel voice conversion using cycle-consistent adversarial networks. Proceedings of the 2018 26th European Signal Processing Conference (EUSIPCO), Roma, Italy.10.23919/EUSIPCO.2018.8553236 ↗
49. Shi, Y., Bu, H., Xu, X., Zhang, S., and Li, M. (2020). Aishell-3: A multi-speaker mandarin tts corpus and the baselines. arXiv.10.21437/Interspeech.2021-755 ↗
50. M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus
51. Park, D.S., Chan, W., Zhang, Y., Chiu, C.C., Zoph, B., Cubuk, E.D., and Le, Q.V. (2019). Specaugment: A simple data augmentation method for automatic speech recognition. arXiv.10.21437/Interspeech.2019-2680 ↗
52. Kaneko, T., Kameoka, H., Tanaka, K., and Hojo, N. (2021, January 6–11). Maskcyclegan-vc: Learning non-parallel voice conversion with filling in frames. Proceedings of the ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414851 ↗
53. Yu, S., Kwak, S., Jang, H., Jeong, J., Huang, J., Shin, J., and Xie, S. (2025, January 24–28). Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. Proceedings of the Thirteenth International Conference on Learning Representations, Singapore.