MixDiff-TTS: Mixture Alignment and Diffusion Model for Text-to-Speech
Source abstract
Abstract
In recent years, deep-learning-based speech synthesis has garnered substantial attention, achieving remarkable advancements in generating human-like speech. However, synthesized speech often lacks naturalness, primarily because models excessively depend on fine-grained text–speech alignment. To address this issue, we propose MixDiff-TTS, a novel non-autoregressive model. MixDiff-TTS incorporates a linguistic encoder based on a mixture alignment mechanism, which combines word-level hard alignment with phoneme-level soft alignment. This design reduces reliance on fine-grained alignment, enabling the model to handle ambiguous phonetic boundaries more robustly. Additionally, we introduce a Word-to-Phoneme Attention module with a relative position bias mechanism to improve the model’s capacity for processing long text sequences. We evaluate the performance of MixDiff-TTS on the LJSpeech dataset. The experimental results show that MixDiff-TTS scores 0.507 for SSIM (Structural Similarity Index) and 6.652 for MCD (Mel Cepstral Distortion). This suggests that the synthesized speech is closer to real speech in spectral structure and exhibits lower spectral distortion than state-of-the-art baselines (such as FastSpeech2 and DiffSpeech). MixDiff-TTS also achieves a MOS (Mean Opinion Score) of 3.95, which is close to that of real speech. These results indicate that MixDiff-TTS can synthesize speech with high naturalness and quality. Ablation studies demonstrate the effectiveness of our method.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Music and Audio Processing · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer, Electronics and Information, Guangxi University, Nanning 530004, China
References (40)
Show source-provided references
1. A novel method for Mandarin speech synthesis by inserting prosodic structure prediction into Tacotron210.1007/s13042-021-01365-x ↗
2. Intelligent personal assistants: A systematic literature review10.1016/j.eswa.2020.113193 ↗
3. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality10.1109/TPAMI.2024.3356232 ↗
4. Neural network based autonomous control of a speech synthesis system
5. Bazzi, A., Slock, D.T., and Meilhac, L. (2016, January 12–14). Sparse recovery using an iterative Variational Bayes algorithm and application to AoA estimation. Proceedings of the 2016 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), Limassol, Cyprus.10.1109/ISSPIT.2016.7886034 ↗
6. Bazzi, A., Slock, D.T.M., and Meilhac, L. (November, January 29). A Newton-type Forward Backward Greedy method for multi-snapshot compressed sensing. Proceedings of the 2017 51st Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA.10.1109/ACSSC.2017.8335537 ↗
7. Li, N., Liu, S., Liu, Y., Zhao, S., and Liu, M. (February, January 27). Neural speech synthesis with transformer network. Proceedings of the AAAI conference on artificial intelligence, Honolulu, HI, USA.
8. Non-autoregressive fully parallel deep convolutional neural speech synthesis10.1109/TASLP.2022.3156797 ↗
9. Huang, R., Zhao, Z., Liu, H., Liu, J., Cui, C., and Ren, Y. (2022, January 10–14). Prodiff: Progressive fast diffusion model for high-quality text-to-speech. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3547855 ↗
10. Łańcucki, A. (2021, January 6–11). Fastpitch: Parallel text-to-speech with pitch prediction. Proceedings of the ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9413889 ↗
11. Oord, A.v.d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016, January 13–15). WaveNet: A Generative Model for Raw Audio. Proceedings of the 9th ISCA Workshop on Speech Synthesis Workshop (SSW 9), Sunnyvale, CA, USA.
12. Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.Y. (2021, January 4). FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. Proceedings of the International Conference on Learning Representations, Vienna, Austria.
13. Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
14. Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.Y. (2019, January 8–14). FastSpeech: Fast, Robust and Controllable Text to Speech. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
15. Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., and Bengio, S. (2017, January 20–24). Tacotron: Towards End-to-End Speech Synthesis. Proceedings of the Interspeech, Stockholm, Sweden.10.21437/Interspeech.2017-1452 ↗
16. Shen, J., Pang, R., Weiss, R.J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., and Skerrv-Ryan, R. (2018, January 15–20). Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461368 ↗
17. Liu, J., Li, C., Ren, Y., Chen, F., and Zhao, Z. (March, January 22). Diffsinger: Singing voice synthesis via shallow diffusion mechanism. Proceedings of the AAAI Conference on Artificial Intelligence, Online.
18. PortaSpeech: Portable and High-Quality Generative Text-to-Speech
19. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
20. Chen, N., Zhang, Y., Zen, H., Weiss, R.J., Norouzi, M., and Chan, W. (2021, January 4). WaveGrad: Estimating Gradients for Waveform Generation. Proceedings of the International Conference on Learning Representations, Vienna, Austria.
21. Kim, H., Kim, S., and Yoon, S. (2022, January 17–23). Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance. Proceedings of the 39th International Conference on Machine Learning, PMLR, Baltimore, MD, USA.
22. Yang, X., Zhou, D., Feng, J., and Wang, X. (2023, January 11–15). Diffusion Probabilistic Model Made Slim. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA.10.1109/CVPR52729.2023.02160 ↗
23. Song, J., Meng, C., and Ermon, S. (2021, January 4). Denoising Diffusion Implicit Models. Proceedings of the International Conference on Learning Representations, Vienna, Austria.
24. Diffusion Models: A Comprehensive Survey of Methods and Applications10.1145/3554729 ↗
25. SinDiffusion: Learning a Diffusion Model From a Single Natural Image10.1109/TPAMI.2025.3532956 ↗
26. Denoising Diffusion Probabilistic Models
27. Prenger, R., Valle, R., and Catanzaro, B. (2019, January 12–17). Waveglow: A Flow-based Generative Network for Speech Synthesis. Proceedings of the ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683143 ↗
28. Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., Driessche, G., Lockhart, E., Cobo, L., and Stimberg, F. (2018, January 10–15). Parallel WaveNet: Fast High-Fidelity Speech Synthesis. Proceedings of the 35th International Conference on Machine Learning, PMLR, Stockholm, Sweden.
29. Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W.Z., Sotelo, J., de Brébisson, A., Bengio, Y., and Courville, A.C. (2019, January 8–14). MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
30. Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. (2021, January 4). DiffWave: A Versatile Diffusion Model for Audio Synthesis. Proceedings of the International Conference on Learning Representations, Vienna, Austria.
31. Huang, R., Lam, M.W.Y., Wang, J., Su, D., Yu, D., Ren, Y., and Zhao, Z. (2022, January 23–29). FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, Vienna, Austria.10.24963/ijcai.2022/577 ↗
32. Rethage, D., Pons, J., and Serra, X. (2018, January 15–20). A Wavenet for Speech Denoising. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462417 ↗
33. Valentini-Botinhao, C., and King, S. (September, January 30). Detection and Analysis of Attention Errors in Sequence-to-Sequence Text-to-Speech. Proceedings of the Interspeech, Brno, Czechia.
34. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA.
35. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
36. Kubichek, R. (1993, January 19–21). Mel-cepstral distance measure for objective speech quality assessment. Proceedings of the IEEE Pacific Rim Conference on Communications Computers and Signal Processing, Victoria, BC, Canada.
37. Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. (2019, January 15–19). Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms. Proceedings of the Interspeech 2019, Graz, Austria.10.21437/Interspeech.2019-2219 ↗
38. Learning to Maximize Speech Quality Directly Using MOS Prediction for Neural Text-to-Speech10.1109/ACCESS.2022.3175810 ↗
39. Langlois, Q., and Jodogne, S. (2023, January 5–7). Practical Study of Deep Learning Models for Speech Synthesis. Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, New York, NY, USA.10.1145/3594806.3596536 ↗
40. Müller, M. (2007). Dynamic time warping. Information Retrieval for Music and Motion, Springer.10.1007/978-3-540-74048-3 ↗