MusicDiffusionNet: Enhancing Text-to-Music Generation with Adaptive Style and Multi-Scale Temporal Mixup Strategies
Source abstract
Abstract
Text-to-music generation aims to automatically produce audio content with semantic consistency and coherent musical structure based on natural language descriptions. However, existing methods still face challenges in terms of style diversity, rhythmic consistency, and long-term structural modeling. To address these issues, we propose a novel text-to-music generation model, termed MusicDiffusionNet (MDN), which integrates diffusion models with the WaveNet architecture to jointly model musical semantics and temporal structure in a continuous latent space. By decoupling high-level semantic conditioning from low-level audio generation, MDN enhances its ability to model long-range musical structure while improving semantic alignment between text and generated music with stable generation behavior. Building upon this framework, we further design two complementary mixing strategies to improve generation quality and structural coherence. Adaptive Style Mixing (ASM) performs weighted interpolation among stylistically similar music samples in the style embedding space, incorporating key and harmonic compatibility constraints to expand the style distribution while avoiding dissonance. Multi-scale Temporal Mixing (MTM) adopts beat-aware temporal decomposition, mixing, and reorganization across multiple time scales, thereby enhancing the modeling of both local and global temporal variations while preserving rhythmic periodicity and musical groove. Both strategies are integrated into the diffusion process as conditional augmentation mechanisms, contributing to improved learning stability and representational capacity under limited data conditions. Experimental results on the Audiostock dataset demonstrate that MDN and its mixing strategies achieve consistent improvements across multiple objective metrics, including generation quality, style diversity, and rhythmic coherence, validating the effectiveness of the proposed approach for text-to-music generation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music Technology and Sound Studies · Generative Adversarial Networks and Image Synthesis · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Modern Science and Technology, China Jiliang University, Hangzhou 310018, China
School of Energy & Environmental Engineering, Hebei University of Technology, Tianjin 300130, China
Funding
General research projects of Zhejiang Provincial Department of Education
References (29)
Show source-provided references
1. The Analysis of Multi-Track Music Generation With Deep Learning Models in Music Production Process10.1109/ACCESS.2024.3439989 ↗
2. Briot, J.P., Hadjeres, G., and Pachet, F.D. (2020). Deep Learning Techniques for Music Generation, Springer.10.1007/978-3-319-70163-9 ↗
3. A survey on generative diffusion models10.1109/TKDE.2024.3361474 ↗
4. A survey on video diffusion models10.1145/3614425 ↗
5. Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., and Dubnov, S. (2023). Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 4–10 June 2023, IEEE.10.1109/ICASSP49357.2023.10095969 ↗
6. Huang, Q., Park, D.S., Wang, T., Denk, T.I., Ly, A., Chen, N., Zhang, Z., Zhang, Z., Yu, J., and Frank, C. (2023). Noise2music: Text-conditioned music generation with diffusion models. arXiv.
7. Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. (2023). Diffusion art or digital forgery? Investigating data replication in diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023, IEEE.10.1109/CVPR52729.2023.00586 ↗
8. Rage against the machine: Copyright infringement in AI-generated music
9. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022, IEEE.10.1109/CVPR52688.2022.01042 ↗
10. Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). Wavenet: A generative model for raw audio. arXiv.
11. Scirea, M., Barros, G.A., Shaker, N., and Togelius, J. (July, January 29). SMUG: Scientific Music Generator. Proceedings of the ICCC, Park City, UT, USA.
12. Music generation with markov models
13. Experiments in musical intelligence (EMI): Non-linear linguistic-based composition
14. A hierarchical recurrent neural network for symbolic melody generation10.1109/TCYB.2019.2953194 ↗
15. Choi, K., Fazekas, G., and Sandler, M. (2016). Text-based LSTM networks for automatic music composition. arXiv.
16. Generative adversarial networks10.1145/3422622 ↗
17. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
18. Schneider, F., Kamal, O., Jin, Z., and Schölkopf, B. (2024, January 11–16). Moûsai: Efficient text-to-music diffusion models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand.10.18653/v1/2024.acl-long.437 ↗
19. Liu, H., Huang, R., Liu, Y., Cao, H., Wang, J., Cheng, X., Zheng, S., and Zhao, Z. (2024). AudioLCM: Text-to-Audio Generation with Latent Consistency Models. arXiv.
20. Lanzendörfer, L.A., Lu, T., Perraudin, N., Herremans, D., and Wattenhofer, R. (2025). Coarse-to-Fine Text-to-Music Latent Diffusion. Proceedings of the ICASSP 2025—2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India, 6–11 April 2025, IEEE.10.1109/ICASSP49660.2025.10888056 ↗
21. Kim, S., Kim, G., Yagishita, S., Han, D., Im, J., and Sung, Y. (2025). Enhancing Diffusion-Based Music Generation Performance with LoRA. Appl. Sci., 15.10.3390/app15158646 ↗
22. A generalization of the beta distribution with applications10.1016/0304-4076(94)01612-4 ↗
23. Oord, A., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., Driessche, G., Lockhart, E., Cobo, L., and Stimberg, F. (2018). Parallel wavenet: Fast high-fidelity speech synthesis. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018, PMLR.
24. Song, K., Zhang, Y., Lei, Y., Cong, J., Li, H., Xie, L., He, G., and Bai, J. (2023). DSPGAN: A gan-based universal vocoder for high-fidelity tts by time-frequency domain supervision from dsp. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 4–10 June 2023, IEEE.10.1109/ICASSP49357.2023.10095105 ↗
25. Soundstream: An end-to-end neural audio codec10.1109/TASLP.2021.3129994 ↗
26. Musical genre classification of audio signals10.1109/TSA.2002.800560 ↗
27. Choi, K., Fazekas, G., Sandler, M., and Cho, K. (2017). Convolutional recurrent neural networks for music classification. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 5–9 March 2017, IEEE.10.1109/ICASSP.2017.7952585 ↗
28. Chen, K., Wu, Y., Liu, H., Nezhurina, M., Berg-Kirkpatrick, T., and Dubnov, S. (2024). Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea, 14–19 April 2024, IEEE.10.1109/ICASSP48485.2024.10447265 ↗
29. Kleijn, W.B., Lim, F.S., Luebs, A., Skoglund, J., Stimberg, F., Wang, Q., and Walters, T.C. (2018). Wavenet based low rate speech coding. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 15–20 April 2018, IEEE.10.1109/ICASSP.2018.8462529 ↗