journal article·2025-02-18Open access

RL–Fusion: The Large Language Model Fusion Method Based on Reinforcement Learning for Task Enhancing

Zijian Wang, Jiayong Li, Yu Liu, Xuhang Li, Cairong Yan, Yanting Zhang

Applied Sciences· Volume 15· Issue 4

Abstract

Model fusion is a technique of growing interest in the field of machine learning, which constructs a generalized model by merging the parameters of multiple independent models with different capabilities without the need to access the original training data or perform costly computations. However, during model fusion, when the number of parameters in a large language model is high, the dimension of the parameter space increases, which makes it more challenging to find the optimal combination of weights. Meanwhile, there is considerable potential for further development in sustainable optimization schemes for task-specific performance enhancement through model fusion in this area. In this paper, we propose a large-scale language model fusion approach based on task-enhanced reinforcement learning (RL–Fusion) to efficiently explore and optimize model fusion configurations. The key innovation of RL–Fusion lies in its use of reinforcement learning to guide parameter selection during model fusion, enabling a more intelligent and adaptive exploration of the parameter space. Additionally, RL–Fusion introduces a dynamic evaluation mechanism that adjusts the evaluation dataset in real-time based on feedback from SOTA models, ensuring continuous enhancement of domain-specific capabilities. RL–Fusion outperforms the baseline model by improving 1.75% in the MMLU benchmark test, 1.8% in the C-eval test, and 1.8% in the Chinese Named Entity Recognition (NER) test on the Yayi NER dataset by 16%. The results show that RL–Fusion is an effective and scalable model fusion solution that improves performance without increasing the computational cost of traditional optimization methods and has a wide range of applications in AI research and practice.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Online Learning and Analytics

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

School of Computer Science and Technology, Donghua University, Shanghai 201620, China

Funding

National Natural Science Foundation of China · Shanghai Sailing Program

References (37)

Show source-provided references
1. Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., and Dong, Z. (2024). A Survey of Large Language Models. arXiv.
2. Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv.
3. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.D.L., Hendricks, L.A., Welbl, J., and Clark, A. (2022). Training Compute-Optimal Large Language Models. arXiv.
4. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models are Few-Shot Learners. arXiv.
5. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv.
6. Yang, E., Shen, L., Guo, G., Wang, X., Cao, X., Zhang, J., and Tao, D. (2024). Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities. arXiv.
7. Ortiz-Jimenez, G., Favero, A., and Frossard, P. (2023). Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. arXiv.
8. Avrahami, O., Lischinski, D., and Fried, O. (2022). Gan Cocktail: Mixing GANs Without Dataset Access, Springer.10.1007/978-3-031-20050-2_13 ↗
9. Nguyen, D., Nguyen, T., Nguyen, K., Phung, D., Bui, H., and Ho, N. (2023). On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks. arXiv.10.1109/ICASSP49357.2023.10097156 ↗
10. Akiba, T., Shing, M., Tang, Y., Sun, Q., and Ha, D. (2024). Evolutionary Optimization of Model Merging Recipes. arXiv.10.1038/s42256-024-00975-8 ↗
11. Liu, D., Wang, Z., Wang, B., Chen, W., Li, C., Tu, Z., Chu, D., Li, B., and Sui, D. (2024). Checkpoint Merging via Bayesian Optimization in LLM Pretraining. arXiv.
12. Zhou, Z., Yang, Y., Yang, X., Yan, J., and Hu, W. (2023). Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity. arXiv.
13. Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal, M. (2023). TIES-Merging: Resolving Interference When Merging Models. arXiv.
14. Aiello, E., Yu, L., Nie, Y., Aghajanyan, A., and Oguz, B. (2023). Jointly Training Large Autoregressive Multimodal Models. arXiv.
15. Ku, L.-W., Martins, A., and Srikumar, V. (2024, January 11–16). Model Composition for Multimodal Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand.
16. Sung, Y.-L., Li, L., Lin, K., Gan, Z., Bansal, M., and Wang, L. (2023). An Empirical Study of Multimodal Model Merging. arXiv.10.18653/v1/2023.findings-emnlp.105 ↗
17. Shukor, M., Dancette, C., Rame, A., and Cord, M. (2023). UnIVAL: Unified Model for Image, Video, Audio and Language Tasks. arXiv.
18. Steps toward Artificial Intelligence10.1109/JRPROC.1961.287775 ↗
19. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring Massive Multitask Language Understanding. arXiv.
20. Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., and Lei, J. (2023). C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. arXiv.
21. Luo, Y., Kong, Q., Xu, N., Cao, J., Hao, B., Qu, B., Chen, B., Zhu, C., Zhao, C., and Zhang, D. (2023). YAYI 2: Multilingual Open-Source Large Language Models. arXiv.
22. Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V., Benedict, B., McQuade, M., and Solawetz, J. (2024). Arcee’s MergeKit: A Toolkit for Merging Large Language Models. arXiv.10.18653/v1/2024.emnlp-industry.36 ↗
23. Tang, A., Shen, L., Luo, Y., Hu, H., Du, B., and Tao, D. (2024). FusionBench: A Comprehensive Benchmark of Deep Model Fusion. arXiv.
24. Singh, S.P., and Jaggi, M. (2023). Model Fusion via Optimal Transport. arXiv.
25. Tatro, N.J., Chen, P.-Y., Das, P., Melnyk, I., Sattigeri, P., and Lai, R. (2020). Optimizing Mode Connectivity via Neuron Alignment. arXiv.
26. Xu, Z., Yuan, K., Wang, H., Wang, Y., Song, M., and Song, J. (2024). Training-Free Pretrained Model Merging. arXiv.10.1109/CVPR52733.2024.00565 ↗
27. Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B. (2023). Repair: Renormalizing Permuted Activations for Interpolation Repair. arXiv.
28. Crisostomi, D., Fumero, M., Baieri, D., Bernard, F., and Rodolà, E. (2024). C2M3: Cycle-Consistent Multi-Model Merging. arXiv.
29. Ramé, A., Couairon, G., Shukor, M., Dancette, C., Gaya, J.-B., Soulier, L., and Cord, M. (2023). Rewarded soups: Towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. arXiv.
30. Shoemake, K. (1985). Animating rotation with quaternion curves. SIGGRAPH ’85: Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques, Association for Computing Machinery.10.1145/325334.325242 ↗
31. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., and Desmaison, A. (2017, January 4–9). Automatic differentiation in PyTorch. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
32. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv.
33. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., and Fan, A. (2024). The Llama 3 Herd of Models. arXiv.
34. Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D.D.L., Hanna, E.B., and Bressand, F. (2024). Mixtral of Experts. arXiv.
35. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., and Huang, F. (2023). Qwen Technical Report. arXiv.
36. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., and Grella, M. (2023). RWKV: Reinventing RNNs for the Transformer Era. arXiv.10.18653/v1/2023.findings-emnlp.936 ↗
37. Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, É., Hesslow, D., Launay, J., and Malartic, Q. (2023). The Falcon Series of Open Language Models. arXiv.