ECE-TTS: A Zero-Shot Emotion Text-to-Speech Model with Simplified and Precise Control
Source abstract
Abstract
Significant advances have been made in emotional speech synthesis technology; however, existing models still face challenges in achieving fine-grained emotion style control and simple yet precise emotion intensity regulation. To address these issues, we propose Easy-Control Emotion Text-to-Speech (ECE-TTS), a zero-shot TTS model built upon the F5-TTS architecture, simplifying emotion modeling while maintaining accurate control. ECE-TTS leverages pretrained emotion recognizers to extract Valence, Arousal, and Dominance (VAD) values, transforming them into Emotion-Adaptive Spherical Vectors (EASV) for precise emotion style representation. Emotion intensity modulation is efficiently realized via simple arithmetic operations on emotion vectors without introducing additional complex modules or training extra regression networks. Emotion style control experiments demonstrate that ECE-TTS achieves a Word Error Rate (WER) of 13.91%, an Aro-Val-Domin SIM of 0.679, and an Emo SIM of 0.594, surpassing GenerSpeech (WER = 16.34%, Aro-Val-Domin SIM = 0.627, Emo SIM = 0.563) and EmoSphere++ (WER = 15.08%, Aro-Val-Domin SIM = 0.656, Emo SIM = 0.578). Subjective Mean Opinion Score (MOS) evaluations (1–5 scale) further confirm improvements in speaker similarity (3.93), naturalness (3.98), and emotional expressiveness (3.94). Additionally, emotion intensity control experiments demonstrate smooth and precise modulation across varying emotional strengths. These results validate ECE-TTS as a highly effective and practical solution for high-quality, emotion-controllable speech synthesis.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Emotion and Mood Recognition · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture, Beijing 102627, China
National Computer Network Emergency Response Technical Team Coordination Center of China, Beijing 100029, China
References (61)
Show source-provided references
1. Shen, K., Ju, Z., Tan, X., Liu, Y., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. (2023). Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv.
2. Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang, D., Liu, Y., Leng, Y., Song, K., and Tang, S. (2024). Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv.
3. Lee, S.H., Choi, H.Y., Kim, S.B., and Lee, S.W. (2023). Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. arXiv.10.21437/Interspeech.2023-1128 ↗
4. Mehta, S., Tu, R., Beskow, J., Székely, É., and Henter, G.E. (2024, January 14–19). Matcha-TTS: A fast TTS architecture with conditional flow matching. Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10448291 ↗
5. Wang, Y., Zhan, H., Liu, L., Zeng, R., Guo, H., Zheng, J., Zhang, Q., Zhang, X., Zhang, S., and Wu, Z. (2024). Maskgct: Zero-shot text-to-speech with masked generative codec transformer. arXiv.
6. Anastassiou, P., Chen, J., Chen, J., Chen, Y., Chen, Z., Chen, Z., Cong, J., Deng, L., Ding, C., and Gao, L. (2024). Seed-tts: A family of high-quality versatile speech generation models. arXiv.
7. Chen, Y., Niu, Z., Ma, Z., Deng, K., Wang, C., Zhao, J., Yu, K., and Chen, X. (2024). F5-tts: A fairytale that fakes fluent and faithful speech with flow matching. arXiv.
8. Triantafyllopoulos, A., and Schuller, B.W. (2024). Expressivity and speech synthesis. arXiv.
9. Zhao, W., and Yang, Z. (2023). An emotion speech synthesis method based on vits. Appl. Sci., 13.10.3390/app13042225 ↗
10. Jung, W., and Lee, J. (2025, April 27). E3-VITS: Emotional End-to-End TTS with Cross-Speaker Style Transfer. Available online: https://openreview.net/pdf?id=qL47xtuEuv.
11. Casanova, E., Weber, J., Shulby, C.D., Junior, A.C., Gölge, E., and Ponti, M.A. (2022, January 17–23). Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA.
12. Kim, J., Kong, J., and Son, J. (2021, January 18–24). Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. Proceedings of the International Conference on Machine Learning, PMLR, Online.
13. Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech
14. Cho, D.H., Oh, H.S., Kim, S.B., and Lee, S.H. (2024). EmoSphere-TTS: Emotional style and intensity modeling via spherical emotion vector for controllable emotional text-to-speech. arXiv.10.21437/Interspeech.2024-398 ↗
15. Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.Y. (2020). Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv.
16. Peebles, W., and Xie, S. (2023, January 2–6). Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00387 ↗
17. Lee, K., Kim, D.W., Kim, J., and Cho, J. (2024). Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer. arXiv.
18. Eskimez, S.E., Wang, X., Thakker, M., Li, C., Tsai, C.H., Xiao, Z., Yang, H., Zhu, Z., Tang, M., and Tan, X. (2024, January 2–5). E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. Proceedings of the 2024 IEEE Spoken Language Technology Workshop (SLT), Macao, China.10.1109/SLT61566.2024.10832320 ↗
19. Liu, Z., Wang, S., Zhu, P., Bi, M., and Li, H. (2024). E1 tts: Simple and fast non-autoregressive tts. arXiv.10.1109/ICASSP49660.2025.10890230 ↗
20. Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, I.S., and Xie, S. (2023, January 18–22). Convnext v2: Co-designing and scaling convnets with masked autoencoders. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01548 ↗
21. Lee, Y., Rabiee, A., and Lee, S.Y. (2017). Emotional end-to-end neural speech synthesizer. arXiv.
22. Exploring transfer learning for low resource emotional tts
23. Tits, N., Wang, F., Haddad, K.E., Pagel, V., and Dutoit, T. (2019). Visualization and interpretation of latent spaces for controlling expressive speech synthesis through audio analysis. arXiv.10.21437/Interspeech.2019-1426 ↗
24. Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., and Bengio, S. (2017). Tacotron: Towards end-to-end speech synthesis. arXiv.10.21437/Interspeech.2017-1452 ↗
25. Shen, J., Pang, R., Weiss, R.J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., and Skerry-Ryan, R. (2018, January 15–20). Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461368 ↗
26. Skerry-Ryan, R., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R., Clark, R., and Saurous, R.A. (2018, January 10–15). Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden.
27. An overview of affective speech synthesis and conversion in the deep learning era10.1109/JPROC.2023.3250266 ↗
28. Neural style transfer: A review10.1109/TVCG.2019.2921336 ↗
29. Wang, Y., Stanton, D., Zhang, Y., Ryan, R.S., Battenberg, E., Shor, J., Xiao, Y., Jia, Y., Ren, F., and Saurous, R.A. (2018, January 10–15). Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden.
30. Wu, H., Wang, X., Eskimez, S.E., Thakker, M., Tompkins, D., Tsai, C.H., Li, C., Xiao, Z., Zhao, S., and Li, J. (2024, January 2–5). Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-To-Speech. Proceedings of the 2024 IEEE Spoken Language Technology Workshop (SLT), Macao, China.10.1109/SLT61566.2024.10832181 ↗
31. Inoue, S., Zhou, K., Wang, S., and Li, H. (2024, January 14–19). Hierarchical emotion prediction and control in text-to-speech synthesis. Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10445996 ↗
32. Inoue, S., Zhou, K., Wang, S., and Li, H. (2024, January 3–6). Fine-grained quantitative emotion editing for speech generation. Proceedings of the 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Macao, China.10.1109/APSIPAASC63619.2025.10848721 ↗
33. A circumplex model of affect10.1037/h0077714 ↗
34. Dawn of the transformer era in speech emotion recognition: Closing the valence gap10.1109/TPAMI.2023.3263585 ↗
35. Cho, D.H., Oh, H.S., Kim, S.B., and Lee, S.W. (2024). EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector. arXiv.10.1109/TAFFC.2025.3561267 ↗
36. Emotion intensity and its control for emotional voice conversion10.1109/TAFFC.2022.3175578 ↗
37. Parikh, D., and Grauman, K. (2011, January 6–13). Relative attributes. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126281 ↗
38. Zhu, X., Yang, S., Yang, G., and Xie, L. (2019, January 14–18). Controlling emotion strength with relative attribute for end-to-end speech synthesis. Proceedings of the 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Singapore.10.1109/ASRU46091.2019.9003829 ↗
39. Lei, Y., Yang, S., and Xie, L. (2021, January 19–22). Fine-grained emotion strength transfer, control and prediction for emotional speech synthesis. Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.10.1109/SLT48900.2021.9383524 ↗
40. Schnell, B., and Garner, P.N. (2021, January 26–28). Improving emotional TTS with an emotion intensity input from unsupervised extraction. Proceedings of the 11th ISCA Speech Synthesis Workshop, Budapest, Hungary.10.21437/SSW.2021-11 ↗
41. Msemotts: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis10.1109/TASLP.2022.3145293 ↗
42. Speech synthesis with mixed emotions10.1109/TAFFC.2022.3233324 ↗
43. Um, S.Y., Oh, S., Byun, K., Jang, I., Ahn, C., and Kang, H.G. (2020, January 4–8). Emotional speech synthesis with rich and granularized control. Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053732 ↗
44. Im, C.B., Lee, S.H., Kim, S.B., and Lee, S.W. (2022, January 7–13). Emoq-tts: Emotion intensity quantization for fine-grained controllable emotional text-to-speech. Proceedings of the ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Online.10.1109/ICASSP43922.2022.9747098 ↗
45. Guo, Y., Du, C., Chen, X., and Yu, K. (2023, January 4–10). Emodiff: Intensity controllable emotional text-to-speech with soft-label guidance. Proceedings of the ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095621 ↗
46. Kalyan, P., Rao, P., Jyothi, P., and Bhattacharyya, P. (2024, January 1–5). Emotion arithmetic: Emotional speech synthesis via weight space interpolation. Proceedings of the Interspeech 2024, Kos Island, Greece.10.21437/Interspeech.2024-2311 ↗
47. Constants across cultures in the face and emotion10.1037/h0030377 ↗
48. Emotional voice conversion: Theory, databases and esd10.1016/j.specom.2021.11.006 ↗
49. IEMOCAP: Interactive emotional dyadic motion capture database10.1007/s10579-008-9076-6 ↗
50. Nguyen, T.A., Hsu, W.N., d’Avirro, A., Shi, B., Gat, I., Fazel-Zarani, M., Remez, T., Copet, J., Synnaeve, G., and Hassid, M. (2023). Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv.10.21437/Interspeech.2023-1905 ↗
51. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice10.1511/2001.28.344 ↗
52. Ho, J., and Salimans, T. (2023). Classifier-free diffusion guidance. arXiv.
53. Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., and Le, M. (2022). Flow matching for generative modeling. arXiv.
54. Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
55. Neural ordinary differential equations
56. Diffusion models beat gans on image synthesis
57. Liu, S., Su, D., and Yu, D. (2022). Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans. arXiv.
58. Li, W., Minematsu, N., and Saito, D. (2024, January 2–5). WET: A Wav2vec 2.0-Based Emotion Transfer Method for Improving the Expressiveness of Text-to-Speech. Proceedings of the Speech Prosody 2024, Leiden, The Netherlands.10.21437/SpeechProsody.2024-249 ↗
59. Ma, Z., Zheng, Z., Ye, J., Li, J., Gao, Z., Zhang, S., and Chen, X. (2023). emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv.10.18653/v1/2024.findings-acl.931 ↗
60. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023, January 23–29). Robust speech recognition via large-scale weak supervision. Proceedings of the International Conference on Machine Learning, PMLR, Honolulu, HI, USA.
61. Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization. arXiv.