Enhancing Diffusion-Based Music Generation Performance with LoRA
Source abstract
Abstract
Recent advancements in generative artificial intelligence have significantly progressed the field of text-to-music generation, enabling users to create music from natural language descriptions. Despite the success of various models, such as MusicLM, MusicGen, and AudioLDM, the current approaches struggle to capture fine-grained genre-specific characteristics, precisely control musical attributes, and handle underrepresented cultural data. This paper introduces a novel, lightweight fine-tuning method for the AudioLDM framework using low-rank adaptation (LoRA). By updating only selected attention and projection layers, the proposed method enables efficient adaptation to musical genres with limited data and computational cost. The proposed method enhances controllability over key musical parameters such as rhythm, emotion, and timbre. At the same time, it maintains the overall quality of music generation. This paper represents the first application of LoRA in AudioLDM, offering a scalable solution for fine-grained, genre-aware music generation and customization. The experimental results demonstrate that the proposed method improves the semantic alignment and statistical similarity compared with the baseline. The contrastive language–audio pretraining score increased by 0.0498, indicating enhanced text-music consistency. The kernel audio distance score decreased by 0.8349, reflecting improved similarity to real music distributions. The mean opinion score ranged from 3.5 to 3.8, confirming the perceptual quality of the generated music.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Music Technology and Sound Studies · Generative Adversarial Networks and Image Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Major of Computer Science and Engineering, Dongguk University-Seoul, Seoul 04620, Republic of Korea
Department of Computer Science and AI, Dongguk University-Seoul, Seoul 04620, Republic of Korea
Funding
Korean government (MSIT) · Ministry of Education of the Republic of Korea
References (29)
Show source-provided references
1. AI methods in algorithmic composition: A comprehensive survey10.1613/jair.3908 ↗
2. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. arXiv.10.1109/CVPR52688.2022.01042 ↗
3. Agostinelli, A., Copet, J., Guitart, A., and Copet, V. (2023). MusicLM: Generating music from text. arXiv.
4. Copet, J., Copet, V., Agostinelli, A., and Défossez, A. (2023). Simple and controllable music generation. arXiv.
5. Liu, H., Huang, R., Cao, H., Wang, J., and Yang, Y. (2023). AudioLDM: Text-to-audio generation with latent diffusion models. arXiv.
6. Piskopani, A.M., Chamberlain, A., and Holter, C.T. (2023, January 11–12). Responsible AI and the arts: The ethical and legal implications of AI in the arts and creative industries. Proceedings of the First International Symposium on Trustworthy Autonomous Systems, Edinburgh, UK.10.1145/3597512.3597528 ↗
7. Ji, S., Luo, J., and Yang, X. (2020). A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions. arXiv.
8. Silva, M.G., Almeida, J.C., Koerich, A.L., and Batista, G.E. (2021). On generalist and domain-specific music classification models and their impacts on Brazilian music genre recognition. J. Braz. Comput. Soc.10.5753/sbcm.2021.19427 ↗
9. Huang, Y.S., Agostinelli, A., Copet, J., and Défossez, A. (2023). Text-to-music generation with controllable emotion conditioning. arXiv.
10. Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv.
11. Fu, Z., Yang, H., So, A.M.C., Lam, W., Bing, L., and Collier, N. (2022). On the Effectiveness of Parameter-Efficient Fine-Tuning. arXiv.
12. Li, X.L., and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv.10.18653/v1/2021.acl-long.353 ↗
13. Lester, B., Al-Rfou, R., and Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv.10.18653/v1/2021.emnlp-main.243 ↗
14. Zaken, E.B., Ravfogel, S., and Goldberg, Y. (2021). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. arXiv.
15. Xu, H., Xu, Y., Nallapati, R., and Xiang, B. (2023). AdaLoRA: Towards Fine-tuning Adaptation of Pretrained Language Models with Low-rank Updates. arXiv.
16. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Quantization-aware Low-Rank Adapter Tuning for Language Generation. arXiv.
17. Wu, Y., Shi, Y., Wei, J., Sun, C., Yang, Y., and Shen, H.T. (2024). DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion. arXiv.
18. Zhu, M., Chen, X., Wang, Z., Yu, B., Zhao, H., and Jia, J. (2025). Modular Customization of Diffusion Models via Blockwise-Parameterized Low-Rank Adaptation. arXiv.
19. Choi, D., Im, J., and Sung, Y. (2024). LoRA Fusion: Enhancing Image Generation. Mathematics, 12.10.3390/math12223474 ↗
20. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
21. He, P., Liu, X., Gao, J., and Chen, W. (2020). DeBERTa: Decoding-enhanced BERT with Disentangled Attention. arXiv.
22. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2025, July 01). Language Models are Unsupervised Multitask Learners. Technical Report, OpenAI. Available online: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
23. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models are Few-Shot Learners. arXiv.
24. Elizalde, B., Khattar, S., Harwath, D., Gan, C., Gygli, M., and Mohamed, A. (2023, January 4–10). CLAP: Learning Audio Concepts from Natural Language Supervision. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095889 ↗
25. Chung, Y., Eu, P., Lee, J., Choi, K., Nam, J., and Chon, B.S. (2025). KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generation. arXiv.
26. ITU-T (1996). Methods for Subjective Determination of Transmission Quality, International Telecommunication Union. Technical Report P.800.
27. Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. (2019). Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms. arXiv.10.21437/Interspeech.2019-2219 ↗
28. A Kernel Two-Sample Test
29. Zen, H., Dang, Q., and Clark, R. (2019, January 15–19). LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. Proceedings of the Interspeech, Graz, Austria.10.21437/Interspeech.2019-2441 ↗