Analysis of Model Merging Methods for Continual Updating of Foundation Models in Distributed Data Settings
Source abstract
Abstract
Foundation models have achieved remarkable success across various domains, but still face critical challenges such as limited data availability, high computational requirements, and rapid knowledge obsolescence. To address these issues, we propose a novel framework that integrates model merging with federated learning to enable continual foundation model updates without centralizing sensitive data. In this framework, each client fine-tunes a local model, and the server merges these models using multiple merging strategies. We experimentally evaluate the effectiveness of these methods using the CLIP model for image classification tasks across diverse datasets. The results demonstrate that advanced merging methods can surpass simple averaging in terms of accuracy, although they introduce challenges such as catastrophic forgetting and sensitivity to hyperparameters. This study defines a realistic and practical problem setting for decentralized foundation model updates, and provides a comparative analysis of merging techniques, offering valuable insights for scalable and privacy-preserving model evolution in dynamic environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Sensor-Based Localization · Geographic Information Systems Studies · Scientific Computing and Data Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan
Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan
Funding
JSPS KAKENHI
References (57)
Show source-provided references
1. Jaiswal, A., Babu, A.R., Zadeh, M.Z., Banerjee, D., and Makedon, F. (2020). A survey on contrastive self-supervised learning. Technologies, 9.10.3390/technologies9010002 ↗
2. Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., and Brunskill, E. (2021). On the opportunities and risks of foundation models. arXiv.
3. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
4. Liu, H., Li, C., Wu, Q., and Lee, Y.J. (2024, January 9–15). Visual instruction tuning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
5. Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., and Millican, K. (2023). Gemini: A family of highly capable multimodal models. arXiv.
6. Li, J., Li, D., Xiong, C., and Hoi, S. (2022, January 17–23). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. Proceedings of the International Conference on Machine Learning (ICML), Baltimore, MD, USA.
7. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning (ICML), Virtual.
8. Jin, Y., Li, J., Liu, Y., Gu, T., Wu, K., Jiang, Z., He, M., Zhao, B., Tan, X., and Gan, Z. (2024). Efficient multimodal large language models: A survey. arXiv.
9. Ren, C., Yu, H., Peng, H., Tang, X., Zhao, B., Yi, L., Tan, A.Z., Gao, Y., Li, A., and Li, X. (2024). Advances and Open Challenges in Federated Foundation Models. arXiv.10.1109/COMST.2025.3552524 ↗
10. Zhuang, W., Chen, C., and Lyu, L. (2023). When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv.
11. Villalobos, P., Sevilla, J., Heim, L., Besiroglu, T., Hobbhahn, M., and Ho, A. (2022). Will we run out of data? An analysis of the limits of scaling datasets in machine learning. arXiv.
12. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv.
13. Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y., and Ghassemi, M. (2024, January 9–15). Aging with grace: Lifelong model editing with discrete key-value adaptors. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
14. Yang, E., Shen, L., Guo, G., Wang, X., Cao, X., Zhang, J., and Tao, D. (2024). Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv.
15. Li, W., Peng, Y., Zhang, M., Ding, L., Hu, H., and Shen, L. (2023). Deep model fusion: A survey. arXiv.
16. Dziadzio, S., Udandarao, V., Roth, K., Prabhu, A., Akata, Z., Albanie, S., and Bethge, M. (2024). How to Merge Your Multimodal Models Over Time?. arXiv.
17. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2017, January 20–22). Communication-efficient learning of deep networks from decentralized data. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Fort Lauderdale, FL, USA.
18. Recent advances of foundation language models-based continual learning: A survey
19. Overcoming catastrophic forgetting in neural networks10.1073/pnas.1611835114 ↗
20. Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. (2019, January 8–14). Experience replay for continual learning. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
21. Gui, Z., Sun, S., Li, R., Yuan, J., An, Z., Roth, K., Prabhu, A., and Torr, P. (2024). kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies. arXiv.
22. Roth, K., Udandarao, V., Dziadzio, S., Prabhu, A., Cherti, M., Vinyals, O., Hénaff, O., Albanie, S., Bethge, M., and Akata, Z. (2024). A Practitioner’s Guide to Continual Multimodal Pretraining. arXiv.
23. Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., and Beyer, L. (2021). How to train your vit? data, augmentation, and regularization in vision transformers. arXiv.
24. Zhang, G., Wang, L., Kang, G., Chen, L., and Wei, Y. (2023, January 2–3). Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01754 ↗
25. Advances and open problems in federated learning10.1561/2200000083 ↗
26. Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2020, January 2–4). Federated optimization in heterogeneous networks. Proceedings of the Machine Learning and Systems, Austin, TX, USA.
27. Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated learning with non-iid data. arXiv.
28. FedBERT: When federated learning meets pre-training10.1145/3510033 ↗
29. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv.
30. Babakniya, S., Elkordy, A.R., Ezzeldin, Y.H., Liu, Q., Song, K.B., El-Khamy, M., and Avestimehr, S. (2023). SLoRA: Federated parameter efficient fine-tuning of language models. arXiv.
31. Zhao, H., Du, W., Li, F., Li, P., and Liu, G. (2023, January 4–10). Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning. Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Island, Greece.10.1109/ICASSP49357.2023.10095356 ↗
32. Zhu, H., Togo, R., Ogawa, T., and Haseyama, M. (2024, January 14–19). Prompt-based personalized federated learning for medical visual question answering. Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10445933 ↗
33. Ensemble learning
34. Wortsman, M., Ilharco, G., Gadre, S.Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A.S., Namkoong, H., Farhadi, A., Carmon, Y., and Kornblith, S. (2022, January 17–23). Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. Proceedings of the International Conference on Machine Learning (ICML), Baltimore, MD, USA.
35. Ramé, A., Ahuja, K., Zhang, J., Cord, M., Bottou, L., and Lopez-Paz, D. (2023, January 23–29). Model ratatouille: Recycling diverse models for out-of-distribution generalization. Proceedings of the International Conference on Machine Learning (ICML), Honolulu, HI, USA.
36. Ilharco, G., Wortsman, M., Gadre, S.Y., Song, S., Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, L. (December, January 28). Patching open-vocabulary models by interpolating weights. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
37. Matena, M.S., and Raffel, C.A. (December, January 28). Merging models with fisher-weighted averaging. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
38. Frankle, J., Dziugaite, G.K., Roy, D., and Carbin, M. (2020, January 13–18). Linear mode connectivity and the lottery ticket hypothesis. Proceedings of the International Conference on Machine Learning (ICML), Virtual.
39. Ilharco, G., Ribeiro, M.T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. (2022). Editing models with task arithmetic. arXiv.
40. Yadav, P., Tam, D., Choshen, L., Raffel, C.A., and Bansal, M. (2024, January 9–15). Ties-merging: Resolving interference when merging models. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
41. Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. (2024, January 21–27). Language models are super mario: Absorbing abilities from homologous models as a free lunch. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria.
42. Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. (2023). Adamerging: Adaptive model merging for multi-task learning. arXiv.
43. Evolutionary optimization of model merging recipes10.1038/s42256-024-00975-8 ↗
44. Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Lopes, R.G., Hajishirzi, H., Farhadi, A., and Namkoong, H. (, January 18–24). Robust fine-tuning of zero-shot models. Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.
45. Davari, M., and Belilovsky, E. (October, January 29). Model breadcrumbs: Scaling multi-task model merging with sparse masks. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.
46. Marczak, D., Twardowski, B., Trzciński, T., and Cygert, S. (October, January 29). Magmax: Leveraging model merging for seamless continual learning. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.
47. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Lake Tahoe, NV, USA.
48. Krause, J., Stark, M., Deng, J., and Fei-Fei, L. (2013, January 1–8). 3d object representations for fine-grained categorization. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Sydney, Australia.10.1109/ICCVW.2013.77 ↗
49. Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. (2014, January 23–28). Describing textures in the wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.461 ↗
50. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification10.1109/JSTARS.2019.2918242 ↗
51. Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. (August, January 31). The German traffic sign recognition benchmark: A multi-class classification competition. Proceedings of the International Joint Conference on Neural Networks (IJCNN), San Jose, CA, USA.
52. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? the kitti vision benchmark suite. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
53. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
54. Remote sensing image scene classification: Benchmark and state of the art10.1109/JPROC.2017.2675998 ↗
55. Sun database: Exploring a large collection of scene categories10.1007/s11263-014-0748-y ↗
56. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A.Y. (2011, January 12–14). Reading digits in natural images with unsupervised feature learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) Workshop, Granada, Spain.
57. Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S.Q. (2024). Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv.