Enhancing MusicGen with Prompt Tuning
Source abstract
Abstract
Generative AI has been gaining attention across various creative domains. In particular, MusicGen stands out as a representative approach capable of generating music based on text or audio inputs. However, it has limitations in producing high-quality outputs for specific genres and fully reflecting user intentions. This paper proposes a prompt tuning technique that effectively adjusts the output quality of MusicGen without modifying its original parameters and optimizes its ability to generate music tailored to specific genres and styles. Experiments were conducted to compare the performance of the traditional MusicGen with the proposed method and evaluate the quality of generated music using the Contrastive Language-Audio Pretraining (CLAP) and Kullback–Leibler Divergence (KLD) scoring approaches. The results demonstrated that the proposed method significantly improved the output quality and musical coherence, particularly for specific genres and styles. Compared with the traditional model, the CLAP score was increased by 0.1270, and the KLD score was increased by 0.00403 on average. The effectiveness of prompt tuning in optimizing the performance of MusicGen validated the proposed method and highlighted its potential for advancing generative AI-based music generation tools.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Music Technology and Sound Studies · Neuroscience and Music Perception
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Major of Semiconductor Science, College of Natural Science, Dongguk University-Seoul, Seoul 04620, Republic of Korea
Department of Computer Science and Artificial Intelligence, Dongguk University-Seoul, Seoul 04620, Republic of Korea
Funding
Institute of Information & communications Technology Planning & Evaluation (IITP) · Ministry of Education of the Republic of Korea and the National Research Foundation of Korea
References (32)
Show source-provided references
1. Simple and controllable music generation
2. Parker, J.D., Spijkervet, J., Kosta, K., Yesiler, F., Kuznetsov, B., Wang, J.C., Avent, M., Chen, J., and Le, D. (2024, January 14–19). StemGen: A music generation model that listens. Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10446088 ↗
3. Efficient neural music generation
4. Mittal, G., Engel, J., Hawthorne, C., and Simon, I. (2021). Symbolic music generation with diffusion models. arXiv.
5. Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M.D. (2023). Audioldm: Text-to-audio generation with latent diffusion models. arXiv.
6. Melechovsky, J., Guo, Z., Ghosal, D., Majumder, N., Herremans, D., and Poria, S. (2023). Mustango: Toward controllable text-to-music generation. arXiv.10.18653/v1/2024.naacl-long.459 ↗
7. Chen, K., Wu, Y., Liu, H., Nezhurina, M., Berg-Kirkpatrick, T., and Dubnov, S. (2024, January 14–19). Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10447265 ↗
8. Ji, S., Luo, J., and Yang, X. (2020). A comprehensive survey on deep music generation: Multi-level representations, algorithms, evaluations, and future directions. arXiv.
9. Wu, C.W., Gururani, S., Laguna, C., Pati, A., Vidwans, A., and Lerch, A. (2016, January 5–9). Towards the objective assessment of music performances. Proceedings of the International Conference on Music Perception and Cognition (ICMPC), San Francisco, CA, USA.
10. Guo, Y., Liu, Y., Zhou, T., Xu, L., and Zhang, Q. (2023). An automatic music generation and evaluation method based on transfer learning. PLoS ONE, 18.10.1371/journal.pone.0283103 ↗
11. Xiong, Z., Wang, W., Yu, J., Lin, Y., and Wang, Z. (2023). A Comprehensive Survey for Evaluation Methodologies of AI-Generated Music. arXiv.
12. Deng, Y., Xu, Z., Zhou, L., Liu, H., and Huang, A. (2021). Research on AI composition recognition based on music rules. Proceedings of the 8th Conference on Sound and Music Technology: Selected Papers from CSMT, Springer.10.1007/978-981-16-1649-5_16 ↗
13. Schlüter, J., and Grill, T. (2015, January 26–30). Exploring data augmentation for improved singing voice detection with neural networks. Proceedings of the ISMIR, Málaga, Spain.
14. Park, D.S., Chan, W., Zhang, Y., Chiu, C.C., Zoph, B., Cubuk, E.D., and Le, Q.V. (2019). Specaugment: A simple data augmentation method for automatic speech recognition. arXiv.10.21437/Interspeech.2019-2680 ↗
15. Dong, H.W., Hsiao, W.Y., Yang, L.C., and Yang, Y.H. (2018, January 2–7). Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11312 ↗
16. Brunner, G., Konrad, A., Wang, Y., and Wattenhofer, R. (2018). MIDI-VAE: Modeling dynamics and instrumentation of music with applications to style transfer. arXiv.
17. Roberts, A., Engel, J., Raffel, C., Hawthorne, C., and Eck, D. (2018, January 10–15). A hierarchical latent vector model for learning long-term structure in music. Proceedings of the International Conference on Machine Learning, PMLR, Stockholm, Sweden.
18. Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. arXiv.10.18653/v1/2021.emnlp-main.243 ↗
19. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv.
20. GPT understands, too10.1016/j.aiopen.2023.08.012 ↗
21. Liu, X., Ji, K., Fu, Y., Tam, W.L., Du, Z., Yang, Z., and Tang, J. (2021). P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv.10.18653/v1/2022.acl-short.8 ↗
22. Li, X.L., and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv.10.18653/v1/2021.acl-long.353 ↗
23. Choi, D., Im, J., and Sung, Y. (2024). LoRA Fusion: Enhancing Image Generation. Mathematics, 12.10.3390/math12223474 ↗
24. Cho, M., Kim, S., Choi, D., and Sung, Y. (2025). Enhanced BLIP-2 Optimization Using LoRA for Generating Dashcam Captions. Appl. Sci., 15.10.3390/app15073712 ↗
25. Choi, Y., Kim, S., Bassole, Y.C.F., and Sung, Y. (2025). Enhanced retrieval-augmented generation using low-rank adaptation. Appl. Sci., 15.10.3390/app15084425 ↗
26. Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. (2022). High fidelity neural audio compression. arXiv.
27. Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H. (2023, January 4–10). Clap learning audio concepts from natural language supervision. Proceedings of the ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095889 ↗
28. On information and sufficiency10.1214/aoms/1177729694 ↗
29. Loughran, R., Walker, J., O’Neill, M., and O’Farrell, M. (2008). The Use of Mel-Frequency Cepstral Coefficients in Musical Instrument Identification, International Computer Music Association.
30. Muller, M., Kurth, F., and Clausen, M. (2005, January 16–19). Chroma-based statistical audio features for audio matching. Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New Paltz, NY, USA.
31. Musical genre classification of audio signals10.1109/TSA.2002.800560 ↗
32. Liu, Z., Huang, J., Wang, Y., and Chen, T. (1997, January 23–25). Audio feature extraction and analysis for scene classification. Proceedings of the First Signal Processing Society Workshop on Multimedia Signal Processing, Princeton, NJ, USA.